[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:35 dist-test-slave-4g6f krb5kdc[5522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:35 dist-test-slave-4g6f krb5kdc[5522](info): set up 2 sockets
Sep 24 15:27:35 dist-test-slave-4g6f krb5kdc[5522](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 15:27:36.636637 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:36 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:36.752315 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33285
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:36.832168 5535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:36.832358 5535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:36.832379 5535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:36.833520 5535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:36.833561 5535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:36.833576 5535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:36.833630 5535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:36.833648 5535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:36.835090 5535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:33285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:36.835366 5535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:36.835605 5535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:36.840817 5540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:36.840818 5543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:36.840886 5541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:36.840938 5535 server_base.cc:1034] running on GCE node
I20260924 15:27:36.841315 5535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:36.841678 5535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:36.842840 5535 hybrid_clock.cc:648] HybridClock initialized: now 1790263656842840 us; error 58 us; skew 500 ppm
Sep 24 15:27:36 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:36.845247 5535 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:36.845736 5535 webserver.cc:460] Webserver started at http://127.5.98.254:34579/ using document root <none> and password file <none>
I20260924 15:27:36.845983 5535 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:36.846050 5535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:36.846199 5535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:36.847245 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "a386bb75936a4e8d8628e7f7ec7358d1"
format_stamp: "Formatted at 2026-09-24 15:27:36 on dist-test-slave-4g6f"
server_key: "a2799e569b091753813abafa74e7d876"
server_key_iv: "5825700d64a9eeb4e4aa4f4d733fc8ff"
server_key_version: "encryptionkey@0"
I20260924 15:27:36.847635 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "a386bb75936a4e8d8628e7f7ec7358d1"
format_stamp: "Formatted at 2026-09-24 15:27:36 on dist-test-slave-4g6f"
server_key: "a2799e569b091753813abafa74e7d876"
server_key_iv: "5825700d64a9eeb4e4aa4f4d733fc8ff"
server_key_version: "encryptionkey@0"
I20260924 15:27:36.849040 5535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:36.849836 5550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:36.850092 5535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:36.850195 5535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "a386bb75936a4e8d8628e7f7ec7358d1"
format_stamp: "Formatted at 2026-09-24 15:27:36 on dist-test-slave-4g6f"
server_key: "a2799e569b091753813abafa74e7d876"
server_key_iv: "5825700d64a9eeb4e4aa4f4d733fc8ff"
server_key_version: "encryptionkey@0"
I20260924 15:27:36.850293 5535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:36.864790 5535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:36.866546 5535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:36.866701 5535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:36.870855 5535 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:33285
I20260924 15:27:36.870927 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:33285 every 8 connection(s)
I20260924 15:27:36.871306 5535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:36.872339 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:36.874840 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1: Bootstrap starting.
I20260924 15:27:36.875538 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:36.875854 5603 log.cc:824] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:36.876552 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 5535
I20260924 15:27:36.876641 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1: No bootstrap required, opened a new log
I20260924 15:27:36.876673 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:36.877334 5515 external_mini_cluster.cc:1428] Setting key 8853b47cb1233d79ab1090d05ecdf25c
I20260924 15:27:36.878721 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } }
I20260924 15:27:36.878885 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:36.878930 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a386bb75936a4e8d8628e7f7ec7358d1, State: Initialized, Role: FOLLOWER
I20260924 15:27:36.879103 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } }
I20260924 15:27:36.879199 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:36.879242 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:36.879292 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:36.880410 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } }
I20260924 15:27:36.880534 5603 leader_election.cc:302] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a386bb75936a4e8d8628e7f7ec7358d1; no voters:
I20260924 15:27:36.880735 5603 leader_election.cc:288] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:36.880759 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:36.880968 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [term 1 LEADER]: Becoming Leader. State: Replica: a386bb75936a4e8d8628e7f7ec7358d1, State: Running, Role: LEADER
I20260924 15:27:36.881079 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:36.881091 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } }
I20260924 15:27:36.881551 5608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a386bb75936a4e8d8628e7f7ec7358d1. Latest consensus state: current_term: 1 leader_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } } }
I20260924 15:27:36.881647 5608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:36.882048 5613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:36.882721 5613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:36.883420 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a386bb75936a4e8d8628e7f7ec7358d1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33285 } } }
I20260924 15:27:36.883517 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:36.884522 5613 catalog_manager.cc:1347] Generated new cluster ID: d3656a82297c4f8ba92ff52ca77ad68f
I20260924 15:27:36.884574 5613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:36 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:36.894713 5613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:36.895416 5613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:36.897183 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:36.878286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.5.98.254:33285)
0924 15:27:36.878483 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:36.878494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:36.878764 (+ 270us) server_negotiation.cc:407] Connection header received
0924 15:27:36.880970 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:36.880977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:36.881037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:36.881210 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:36.883654 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:36.885321 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:36.887103 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:36.887863 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:36.887978 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:36.892533 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:36.892537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:36.892538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:36.892564 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:36.894816 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:36.895315 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:36.895319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:36.895319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:36.895448 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:36.895825 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:36.895827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:36.895827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:36.896129 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:36.896275 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:36.896770 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 15:27:36.896944 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260924 15:27:36.899258 5613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a386bb75936a4e8d8628e7f7ec7358d1: Generated new TSK 0
I20260924 15:27:36.899405 5613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:36.928272 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:37.013300 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:37.013535 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:37.013573 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:37.014724 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:37.014796 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:37.014844 5629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:37.016358 5629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:37.016621 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:37.016853 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:37.022055 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:37.022135 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.022195 5629 server_base.cc:1034] running on GCE node
W20260924 15:27:37.022055 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.022641 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:37.022884 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:37.024063 5629 hybrid_clock.cc:648] HybridClock initialized: now 1790263657024050 us; error 32 us; skew 500 ppm
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:37.026841 5629 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:37.027323 5629 webserver.cc:460] Webserver started at http://127.5.98.193:36151/ using document root <none> and password file <none>
I20260924 15:27:37.027565 5629 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:37.027630 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:37.027794 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:37.028822 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "15b97065e760422fb55a54b3ea3fd6c8"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "95f97ab50b4108dd3deb0acce493a370"
server_key_iv: "1142a3563e336183898158d533332916"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.029194 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "15b97065e760422fb55a54b3ea3fd6c8"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "95f97ab50b4108dd3deb0acce493a370"
server_key_iv: "1142a3563e336183898158d533332916"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.030586 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:37.031488 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.031690 5629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:37.031771 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "15b97065e760422fb55a54b3ea3fd6c8"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "95f97ab50b4108dd3deb0acce493a370"
server_key_iv: "1142a3563e336183898158d533332916"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.031858 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:37.042176 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:37.044189 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:37.044340 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:37.044559 5629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:37.044858 5629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:37.044912 5629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.044952 5629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:37.045009 5629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.050828 5629 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:43763
I20260924 15:27:37.050899 5758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:43763 every 8 connection(s)
I20260924 15:27:37.051206 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:37.053063 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 5629
I20260924 15:27:37.053174 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:37.053370 5515 external_mini_cluster.cc:1428] Setting key bfd3509f216b22f717c120e6ceb9895a
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
I20260924 15:27:37.061435 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.051949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:60075 (local address 127.5.98.254:33285)
0924 15:27:37.052013 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:37.052015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:37.052246 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:27:37.052794 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:37.052795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:37.052806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:37.052906 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:37.054236 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.054748 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.056129 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.056329 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.056414 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.058532 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:37.058533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:37.058534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:37.058541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:37.060018 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.060423 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.060424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.060424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.060497 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.060748 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.060749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.060749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.060863 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:37.060937 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:37.061248 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 15:27:37.061346 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:37.061589 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.051944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33285 (local address 127.5.98.193:60075)
0924 15:27:37.052129 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:37.052162 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:37.052703 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:37.053026 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:37.053030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:37.053868 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:37.054133 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.054140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.054840 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.054841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.056022 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.056025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.056423 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.056425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.056558 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.057050 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:37.057059 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:37.058427 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:37.060110 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.060117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.060119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.060336 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.060562 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.060563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.060563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.060673 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.060928 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:37.060935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:37.061078 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 15:27:37.061222 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 15:27:37.061376 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260924 15:27:37.062237 5759 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33285
I20260924 15:27:37.062345 5759 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:37.062544 5759 heartbeater.cc:511] Master 127.5.98.254:33285 requested a full tablet report, sending...
I20260924 15:27:37.063210 5567 ts_manager.cc:194] Registered new tserver with Master: 15b97065e760422fb55a54b3ea3fd6c8 (127.5.98.193:43763)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
I20260924 15:27:37.063979 5567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:60075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:37.078233 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:37.161465 5764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:37.161660 5764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:37.161684 5764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:37.162793 5764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:37.162833 5764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:37.162860 5764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:37.164290 5764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:37.164537 5764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:37.164772 5764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:37.169802 5770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.169992 5764 server_base.cc:1034] running on GCE node
W20260924 15:27:37.169814 5772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:37.169803 5769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.170315 5764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:37.170563 5764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:37.171731 5764 hybrid_clock.cc:648] HybridClock initialized: now 1790263657171720 us; error 41 us; skew 500 ppm
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:37.174273 5764 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:37.174696 5764 webserver.cc:460] Webserver started at http://127.5.98.194:38931/ using document root <none> and password file <none>
I20260924 15:27:37.174935 5764 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:37.175010 5764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:37.175166 5764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:37.176273 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "b719832fbc40421ea1b9a6779032841e"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "73c0b10db2a2a7b7155ba87025e5b8e2"
server_key_iv: "00a9a7c5fede562a1ae49a718323c861"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.176630 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "b719832fbc40421ea1b9a6779032841e"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "73c0b10db2a2a7b7155ba87025e5b8e2"
server_key_iv: "00a9a7c5fede562a1ae49a718323c861"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.177943 5764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:37.178697 5779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.178864 5764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:37.178967 5764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "b719832fbc40421ea1b9a6779032841e"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "73c0b10db2a2a7b7155ba87025e5b8e2"
server_key_iv: "00a9a7c5fede562a1ae49a718323c861"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.179056 5764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:37.214816 5764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:37.217003 5764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:37.217216 5764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:37.217516 5764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:37.217856 5764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:37.217911 5764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.217947 5764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:37.218003 5764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.224418 5764 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:39931
I20260924 15:27:37.224550 5893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:39931 every 8 connection(s)
I20260924 15:27:37.224831 5764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:37.233116 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 5764
I20260924 15:27:37.233210 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:37.233397 5515 external_mini_cluster.cc:1428] Setting key 59ea9b2798888d9d3f71825a0fcf92c8
I20260924 15:27:37.233858 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.225529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:54439 (local address 127.5.98.254:33285)
0924 15:27:37.225603 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:37.225605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:37.225814 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:27:37.226294 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:37.226295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:37.226305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:37.226361 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:37.227236 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.227843 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.228791 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.228969 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.229029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.231047 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:37.231048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:37.231048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:37.231054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:37.232378 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.232777 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.232778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.232778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.232827 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.233133 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.233134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.233134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.233293 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:37.233347 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:37.233687 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 15:27:37.233768 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:27:37.234119 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.225487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33285 (local address 127.5.98.194:54439)
0924 15:27:37.225683 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:37.225726 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:37.226215 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:37.226416 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:37.226419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:37.226963 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:37.227164 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.227169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.227919 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.227920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.228722 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.228724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.229047 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.229048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.229156 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.229656 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:37.229671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:37.230963 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:37.232465 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.232471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.232473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.232699 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.232887 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.232888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.232888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.233055 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.233346 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:37.233354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:37.233520 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 15:27:37.233662 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 15:27:37.233824 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260924 15:27:37.234611 5894 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33285
I20260924 15:27:37.234740 5894 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:37.234942 5894 heartbeater.cc:511] Master 127.5.98.254:33285 requested a full tablet report, sending...
I20260924 15:27:37.235325 5567 ts_manager.cc:194] Registered new tserver with Master: b719832fbc40421ea1b9a6779032841e (127.5.98.194:39931)
I20260924 15:27:37.235982 5567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:54439
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:37.258463 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:37.341529 5899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:37.341779 5899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:37.341826 5899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:37.343120 5899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:37.343190 5899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:37.343241 5899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:37.344940 5899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:37.345232 5899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:37.345484 5899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:37.350575 5904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:37.350593 5905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.350761 5899 server_base.cc:1034] running on GCE node
W20260924 15:27:37.350677 5907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:37.351085 5899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:37.351279 5899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:37.352392 5899 hybrid_clock.cc:648] HybridClock initialized: now 1790263657352377 us; error 28 us; skew 500 ppm
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:37.354876 5899 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:37.355293 5899 webserver.cc:460] Webserver started at http://127.5.98.195:40859/ using document root <none> and password file <none>
I20260924 15:27:37.355484 5899 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:37.355525 5899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:37.355609 5899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:37.356614 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "7fd179e1936f474587b7759cf773f5d5"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "a4d6e3cc3f6891b2271fe90ec20b5c28"
server_key_iv: "0172fdacac2f1b5d6a5b11005b499ddd"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.356906 5899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "7fd179e1936f474587b7759cf773f5d5"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "a4d6e3cc3f6891b2271fe90ec20b5c28"
server_key_iv: "0172fdacac2f1b5d6a5b11005b499ddd"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.358132 5899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:37.358811 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.358974 5899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:37.359074 5899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "7fd179e1936f474587b7759cf773f5d5"
format_stamp: "Formatted at 2026-09-24 15:27:37 on dist-test-slave-4g6f"
server_key: "a4d6e3cc3f6891b2271fe90ec20b5c28"
server_key_iv: "0172fdacac2f1b5d6a5b11005b499ddd"
server_key_version: "encryptionkey@0"
I20260924 15:27:37.359151 5899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:37.379973 5899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:37.381779 5899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:37.382042 5899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:37.382223 5899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:37.382570 5899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:37.382624 5899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.382666 5899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:37.382694 5899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:37.389353 5899 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:41479
I20260924 15:27:37.389398 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:41479 every 8 connection(s)
I20260924 15:27:37.389704 5899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:37.392977 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 5899
I20260924 15:27:37.393074 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:37.393240 5515 external_mini_cluster.cc:1428] Setting key 8efcc9e61542bb980d35c324e8217602
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:37.399204 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.390341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:41487 (local address 127.5.98.254:33285)
0924 15:27:37.390384 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:37.390386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:37.390609 (+ 223us) server_negotiation.cc:407] Connection header received
0924 15:27:37.391077 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:37.391078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:37.391087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:37.391145 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:37.392146 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.392635 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.394083 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.394370 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.394435 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.396557 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:37.396558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:37.396558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:37.396565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:37.397828 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.398312 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.398314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.398314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.398371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.398649 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.398650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.398650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.398756 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:37.398813 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:37.399068 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 15:27:37.399138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:37.399580 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.390319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33285 (local address 127.5.98.195:41487)
0924 15:27:37.390484 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:37.390524 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:37.390993 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:37.391233 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:37.391236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:37.391859 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:37.392054 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.392060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.392713 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.392713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.394004 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:37.394007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.394464 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.394465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:37.394574 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.395060 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:37.395082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:37.396472 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:37.397919 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.397925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.397928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.398229 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.398455 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:37.398456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:37.398456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:37.398583 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:37.398818 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:37.398822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:37.398927 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:27:37.399007 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:27:37.399404 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260924 15:27:37.400064 6029 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33285
I20260924 15:27:37.400156 6029 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:37.400285 6029 heartbeater.cc:511] Master 127.5.98.254:33285 requested a full tablet report, sending...
I20260924 15:27:37.400652 5567 ts_manager.cc:194] Registered new tserver with Master: 7fd179e1936f474587b7759cf773f5d5 (127.5.98.195:41479)
I20260924 15:27:37.401088 5567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:41487
I20260924 15:27:37.405097 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:37.427870 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.420250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.5.98.193:43763)
0924 15:27:37.420418 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:37.420422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:37.420432 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:37.420457 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:37.420458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:37.420471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:37.420538 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:37.421045 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.421742 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.422443 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.422760 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.422812 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.425027 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:37.425037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:37.425039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:37.425051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:37.426641 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.427000 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.427004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.427004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.427064 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.427272 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.427273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.427273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.427415 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:37.427588 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:37.427635 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 15:27:37.427735 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
W20260924 15:27:37.428233 5673 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54284
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:37.449304 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:37.442553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.5.98.193:43763)
0924 15:27:37.442617 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:37.442620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:37.442718 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:27:37.442808 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:37.442809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:37.442819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:37.442883 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:37.443411 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.443945 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.444672 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:37.444871 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:37.444939 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:37.447024 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:37.447026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:37.447026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:37.447033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:37.448358 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.448661 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.448662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.448662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.448722 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:37.448892 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:37.448892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:37.448892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:37.448987 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:37.449068 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:37.449149 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:27:37.449208 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:27:37.449858 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 5629
I20260924 15:27:37.454710 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 5764
I20260924 15:27:37.459077 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 5899
I20260924 15:27:37.463321 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 5535
2026-09-24T15:27:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1891 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[6043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[6043](info): set up 2 sockets
Sep 24 15:27:37 dist-test-slave-4g6f krb5kdc[6043](info): commencing operation
krb5kdc: starting...
W20260924 15:27:38.501006 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:38 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:38.616883 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40361
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:38.702327 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:38.702595 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:38.702629 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:38.703773 6056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:38.703842 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:38.703866 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:38.703886 6056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:38.703912 6056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:38.705368 6056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:38.705643 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:38.705873 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:38.710767 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:38.710774 6061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:38.710778 6062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:38.711053 6056 server_base.cc:1034] running on GCE node
I20260924 15:27:38.711251 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:38.711493 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:38.712643 6056 hybrid_clock.cc:648] HybridClock initialized: now 1790263658712630 us; error 28 us; skew 500 ppm
Sep 24 15:27:38 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:38.715108 6056 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:38.715580 6056 webserver.cc:460] Webserver started at http://127.5.98.254:42545/ using document root <none> and password file <none>
I20260924 15:27:38.715840 6056 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:38.715911 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:38.716068 6056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:38.717089 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "707419752d3846568300aa0a918c99c1"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "900dcbbe2b5a6190284c8c34fd14b6f2"
server_key_iv: "50fbb54c0b7f0ed32039c83969fc2743"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.717447 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "707419752d3846568300aa0a918c99c1"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "900dcbbe2b5a6190284c8c34fd14b6f2"
server_key_iv: "50fbb54c0b7f0ed32039c83969fc2743"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.718767 6056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:38.719516 6071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:38.719738 6056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:38.719834 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "707419752d3846568300aa0a918c99c1"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "900dcbbe2b5a6190284c8c34fd14b6f2"
server_key_iv: "50fbb54c0b7f0ed32039c83969fc2743"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.719919 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:38.733735 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:38.735607 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:38.735803 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:38.739732 6056 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:40361
I20260924 15:27:38.739814 6124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:40361 every 8 connection(s)
I20260924 15:27:38.740113 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:38.740908 6125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:38.741191 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6056
I20260924 15:27:38.741282 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:38.741466 5515 external_mini_cluster.cc:1428] Setting key ba27e19401704bba0266a61ed73e9cd8
I20260924 15:27:38.743460 6125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: Bootstrap starting.
I20260924 15:27:38.744088 6125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:38.744383 6125 log.cc:824] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:38.745158 6125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: No bootstrap required, opened a new log
I20260924 15:27:38.747680 6125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } }
I20260924 15:27:38.747862 6125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:38.747896 6125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707419752d3846568300aa0a918c99c1, State: Initialized, Role: FOLLOWER
I20260924 15:27:38.748052 6125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } }
I20260924 15:27:38.748129 6125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:38.748152 6125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:38.748214 6125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:27:38 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:38.748780 6125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } }
I20260924 15:27:38.748909 6125 leader_election.cc:302] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707419752d3846568300aa0a918c99c1; no voters:
I20260924 15:27:38.749104 6125 leader_election.cc:288] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:38.749356 6125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:38.749442 6130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:38.749606 6130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [term 1 LEADER]: Becoming Leader. State: Replica: 707419752d3846568300aa0a918c99c1, State: Running, Role: LEADER
I20260924 15:27:38.749699 6130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } }
I20260924 15:27:38.750510 6131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707419752d3846568300aa0a918c99c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } } }
I20260924 15:27:38.750612 6131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:38.750877 6132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707419752d3846568300aa0a918c99c1. Latest consensus state: current_term: 1 leader_uuid: "707419752d3846568300aa0a918c99c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707419752d3846568300aa0a918c99c1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40361 } } }
I20260924 15:27:38.750977 6132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:38.751135 6144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:38.751183 6144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:38.751338 6145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:38.751542 6145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:38.752722 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:38.741981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.5.98.254:40361)
0924 15:27:38.742172 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:38.742187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:38.742213 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:27:38.742639 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:38.742645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:38.742690 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:38.742839 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:38.743461 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.744596 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.745628 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.746004 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.746097 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:38.748886 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:38.748890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:38.748891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:38.748903 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:38.751007 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:38.751422 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:38.751425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:38.751425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:38.751527 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:38.751804 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:38.751805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:38.751805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:38.752092 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:38.752225 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:38.752378 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:27:38.752511 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 15:27:38.753098 6145 catalog_manager.cc:1347] Generated new cluster ID: 1ec0215a0f6c44789cc570943440bb67
I20260924 15:27:38.753145 6145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:38.788261 6145 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:38.788841 6145 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:38.796334 6145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707419752d3846568300aa0a918c99c1: Generated new TSK 0
I20260924 15:27:38.796531 6145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:38.830307 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:38.913270 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:38.913486 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:38.913511 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:38.914654 6151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:38.914700 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:38.914757 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:38.916365 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:38.916625 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:38.916867 6151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:38.921962 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:38.921993 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:38.922017 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:38.922176 6151 server_base.cc:1034] running on GCE node
I20260924 15:27:38.922441 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:38.922700 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:38.923861 6151 hybrid_clock.cc:648] HybridClock initialized: now 1790263658923823 us; error 53 us; skew 500 ppm
Sep 24 15:27:38 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:38.926260 6151 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:38.926704 6151 webserver.cc:460] Webserver started at http://127.5.98.193:38995/ using document root <none> and password file <none>
I20260924 15:27:38.926934 6151 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:38.927011 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:38.927166 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:38.928248 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "6364e7dce71649d8b07ee411dfb795b8"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "761fcff7216b8f09252e4c11534e1aec"
server_key_iv: "ac182402642f76c4604118e6c876750b"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.928607 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "6364e7dce71649d8b07ee411dfb795b8"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "761fcff7216b8f09252e4c11534e1aec"
server_key_iv: "ac182402642f76c4604118e6c876750b"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.929911 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:38.930699 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:38.930883 6151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:38.930979 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "6364e7dce71649d8b07ee411dfb795b8"
format_stamp: "Formatted at 2026-09-24 15:27:38 on dist-test-slave-4g6f"
server_key: "761fcff7216b8f09252e4c11534e1aec"
server_key_iv: "ac182402642f76c4604118e6c876750b"
server_key_version: "encryptionkey@0"
I20260924 15:27:38.931067 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:38.941717 6151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:38.943457 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:38.943615 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:38.943850 6151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:38.944155 6151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:38.944208 6151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:38.944240 6151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:38.944268 6151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:38.950222 6151 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:44779
I20260924 15:27:38.950255 6280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:44779 every 8 connection(s)
I20260924 15:27:38.950592 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:38.954741 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6151
I20260924 15:27:38.954852 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:38.955010 5515 external_mini_cluster.cc:1428] Setting key 5c35e5dd0b41a5230f04663b796430c6
Sep 24 15:27:38 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:38.961099 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:38.951402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:45323 (local address 127.5.98.254:40361)
0924 15:27:38.951488 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:38.951491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:38.951594 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:27:38.952527 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:38.952529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:38.952539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:38.952586 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:38.953193 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.953750 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.954779 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.955160 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.955244 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:38.957937 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:38.957939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:38.957939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:38.957948 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:38.959501 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:38.960017 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:38.960018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:38.960019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:38.960091 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:38.960383 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:38.960384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:38.960385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:38.960517 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:38.960595 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:38.960895 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 15:27:38.961003 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:27:38.961272 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:38.951314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40361 (local address 127.5.98.193:45323)
0924 15:27:38.951456 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:38.951529 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:38.952428 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:38.952645 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:38.952648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:38.952882 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:38.953070 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:38.953074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.953850 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.953852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:38.954672 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:38.954674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:38.955262 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:38.955263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:38.955440 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:38.955990 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:38.956001 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:38.957830 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:38.959619 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:38.959627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:38.959629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:38.959921 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:38.960177 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:38.960178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:38.960178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:38.960300 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:38.960600 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:38.960608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:38.960750 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 15:27:38.960858 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:27:38.961023 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 15:27:38.961963 6281 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40361
I20260924 15:27:38.962051 6281 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:38.962262 6281 heartbeater.cc:511] Master 127.5.98.254:40361 requested a full tablet report, sending...
I20260924 15:27:38.962843 6088 ts_manager.cc:194] Registered new tserver with Master: 6364e7dce71649d8b07ee411dfb795b8 (127.5.98.193:44779)
I20260924 15:27:38.963832 6088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:45323
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:38.980561 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:39.061125 6286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:39.061319 6286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:39.061342 6286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:39.062415 6286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:39.062448 6286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:39.062479 6286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:39.063867 6286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:39.064039 6286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:39.064208 6286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:39.069172 6294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:39.069160 6292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:39.069191 6286 server_base.cc:1034] running on GCE node
W20260924 15:27:39.069157 6291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:39.069617 6286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:39.069895 6286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:39.071049 6286 hybrid_clock.cc:648] HybridClock initialized: now 1790263659071052 us; error 44 us; skew 500 ppm
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:39.073582 6286 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:39.073992 6286 webserver.cc:460] Webserver started at http://127.5.98.194:35907/ using document root <none> and password file <none>
I20260924 15:27:39.074213 6286 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:39.074281 6286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:39.074433 6286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:39.075496 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "b66c28418f444406a91bc7e3029461d5"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "fc12583e501a185b67d7337035852d29"
server_key_iv: "ecf5dfa96773bfefa78294993f3ca393"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.075881 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "b66c28418f444406a91bc7e3029461d5"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "fc12583e501a185b67d7337035852d29"
server_key_iv: "ecf5dfa96773bfefa78294993f3ca393"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.077171 6286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:39.077919 6301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.078099 6286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:39.078195 6286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "b66c28418f444406a91bc7e3029461d5"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "fc12583e501a185b67d7337035852d29"
server_key_iv: "ecf5dfa96773bfefa78294993f3ca393"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.078275 6286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:39.093741 6286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:39.096118 6286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:39.096283 6286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:39.096532 6286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:39.096846 6286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:39.096900 6286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.096946 6286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:39.096969 6286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.102679 6286 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46145
I20260924 15:27:39.102728 6415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46145 every 8 connection(s)
I20260924 15:27:39.103032 6286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:39.105289 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6286
I20260924 15:27:39.105367 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:39.105535 5515 external_mini_cluster.cc:1428] Setting key d63872147a3032714dfd195a1faf0703
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:39.112916 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.103803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:57425 (local address 127.5.98.254:40361)
0924 15:27:39.103847 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.103849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.104099 (+ 250us) server_negotiation.cc:407] Connection header received
0924 15:27:39.105069 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.105071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.105082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.105134 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.105876 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.106373 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.107438 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.107704 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.107781 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.109930 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.109931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.109932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.109938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.111473 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.111951 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.111952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.111953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.112020 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.112285 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.112286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.112286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.112397 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.112466 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.112750 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.112827 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:39.113099 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.103795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40361 (local address 127.5.98.194:57425)
0924 15:27:39.103976 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.104017 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.104942 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.105219 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.105228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.105537 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:39.105775 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.105780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.106464 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.106465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.107354 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.107357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.107793 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.107794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.107950 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.108456 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:39.108466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:39.109818 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:39.111574 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:39.111582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:39.111584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:39.111861 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:39.112087 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:39.112088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:39.112089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:39.112200 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:39.112477 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:39.112483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:39.112614 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.112716 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.112869 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260924 15:27:39.113560 6416 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40361
I20260924 15:27:39.113663 6416 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:27:39.113838 6416 heartbeater.cc:511] Master 127.5.98.254:40361 requested a full tablet report, sending...
I20260924 15:27:39.114250 6088 ts_manager.cc:194] Registered new tserver with Master: b66c28418f444406a91bc7e3029461d5 (127.5.98.194:46145)
I20260924 15:27:39.114622 6088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:57425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:39.131614 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:39.217782 6421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:39.218015 6421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:39.218055 6421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:39.219193 6421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:39.219259 6421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:39.219304 6421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:39.220863 6421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:39.221139 6421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:39.221359 6421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:39.226231 6427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:39.226241 6426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:39.226274 6429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:39.226574 6421 server_base.cc:1034] running on GCE node
I20260924 15:27:39.226804 6421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:39.227026 6421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:39.228225 6421 hybrid_clock.cc:648] HybridClock initialized: now 1790263659228205 us; error 29 us; skew 500 ppm
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:39.230647 6421 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:39.231004 6421 webserver.cc:460] Webserver started at http://127.5.98.195:46155/ using document root <none> and password file <none>
I20260924 15:27:39.231223 6421 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:39.231267 6421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:39.231417 6421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:39.232511 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "ae4bebee9f902b332da380cebab5f74c"
server_key_iv: "d61eca6bf9388b939dd5819374caed3e"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.232864 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "ae4bebee9f902b332da380cebab5f74c"
server_key_iv: "d61eca6bf9388b939dd5819374caed3e"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.234258 6421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:27:39.235041 6436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.235249 6421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:39.235354 6421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6"
format_stamp: "Formatted at 2026-09-24 15:27:39 on dist-test-slave-4g6f"
server_key: "ae4bebee9f902b332da380cebab5f74c"
server_key_iv: "d61eca6bf9388b939dd5819374caed3e"
server_key_version: "encryptionkey@0"
I20260924 15:27:39.235450 6421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:39.266819 6421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:39.268923 6421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:39.269093 6421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:39.269357 6421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:39.269682 6421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:39.269737 6421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.269776 6421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:39.269805 6421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:39.275837 6421 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:38311
I20260924 15:27:39.275918 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:38311 every 8 connection(s)
I20260924 15:27:39.276242 6421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:39.276623 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6421
I20260924 15:27:39.276722 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:39.276916 5515 external_mini_cluster.cc:1428] Setting key 8461c1c4b5ba01190789aae4909fdd66
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:39.285784 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.277007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:37323 (local address 127.5.98.254:40361)
0924 15:27:39.277050 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.277052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.277394 (+ 342us) server_negotiation.cc:407] Connection header received
0924 15:27:39.278474 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.278475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.278487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.278532 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.279142 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.279693 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.280726 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.280986 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.281055 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.283087 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.283089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.283089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.283096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.284592 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.284957 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.284958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.284958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.285007 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.285230 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.285231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.285231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.285338 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.285389 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.285662 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.285722 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:39.286002 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.277003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40361 (local address 127.5.98.195:37323)
0924 15:27:39.277251 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.277300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.278369 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.278600 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.278604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.278855 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:39.279051 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.279055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.279786 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.279787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.280626 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.280628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.281064 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.281065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.281203 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.281629 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:39.281636 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:39.282999 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:39.284666 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:39.284672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:39.284673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:39.284878 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:39.285066 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:39.285066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:39.285067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:39.285162 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:39.285400 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:39.285406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:39.285541 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.285628 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.285751 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260924 15:27:39.286545 6551 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40361
I20260924 15:27:39.286654 6551 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:39.286840 6551 heartbeater.cc:511] Master 127.5.98.254:40361 requested a full tablet report, sending...
I20260924 15:27:39.287196 6088 ts_manager.cc:194] Registered new tserver with Master: b756b0d72b6a4a4fb48a9f1585a8c2d6 (127.5.98.195:38311)
I20260924 15:27:39.287532 6088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:37323
I20260924 15:27:39.288669 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:39.295861 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.290199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.5.98.254:40361)
0924 15:27:39.290243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.290245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.290321 (+ 76us) server_negotiation.cc:407] Connection header received
0924 15:27:39.290393 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.290394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.290404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.290466 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.290987 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.291451 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.292214 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.292455 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.292542 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.293579 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.293580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.293581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.293587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.294695 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.295026 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.295027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.295028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.295078 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.295313 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.295314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.295314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.295460 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.295564 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.295689 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.295765 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:39.297909 6088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:39.300473 6088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:39.306969 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.303224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.5.98.193:44779)
0924 15:27:39.303399 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.303404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.303470 (+ 66us) server_negotiation.cc:407] Connection header received
0924 15:27:39.303519 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.303522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.303537 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.303610 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:39.303873 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.304607 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306028 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.306580 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306668 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.306729 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.306848 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.306889 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260924 15:27:39.306959 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.302956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44779 (local address 127.0.0.1:35744)
0924 15:27:39.303377 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.303388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.303455 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.303619 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.303622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.303624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:39.303708 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.303710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.304710 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.304711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.305924 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.305927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.306667 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.306745 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.306758 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.306832 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.306882 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260924 15:27:39.307286 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.302955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:38311 (local address 127.0.0.1:35614)
0924 15:27:39.303228 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.303257 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.303332 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.303639 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.303641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.303642 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:39.303713 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.303715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.305021 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.305022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.306134 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.306136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.307051 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.307052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.307125 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.307129 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.307180 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.307217 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":54,"thread_start_us":56,"threads_started":1}
I20260924 15:27:39.307318 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.303044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.5.98.195:38311)
0924 15:27:39.303408 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.303412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.303422 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:39.303461 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.303463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.303479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.303554 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:39.303793 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.304913 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306219 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.306890 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306949 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.306997 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.307198 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.307246 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":72,"threads_started":1}
I20260924 15:27:39.307796 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.303222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.5.98.194:46145)
0924 15:27:39.303593 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.303597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.303605 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:39.303643 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.303644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.303685 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.303761 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:39.303962 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.304991 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306050 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.306779 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306848 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.306906 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.307638 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.307727 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":77,"threads_started":1}
I20260924 15:27:39.307713 6215 tablet_service.cc:1467] Processing CreateTablet for tablet 962de5e0a0e94fe2b01bfe89c131a2d9 (DEFAULT_TABLE table=test-table [id=a64bd8399d6244baa11c9664a6f4e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:39.308063 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962de5e0a0e94fe2b01bfe89c131a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:39.308010 6485 tablet_service.cc:1467] Processing CreateTablet for tablet 962de5e0a0e94fe2b01bfe89c131a2d9 (DEFAULT_TABLE table=test-table [id=a64bd8399d6244baa11c9664a6f4e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:39.308194 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.302973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46145 (local address 127.0.0.1:36368)
0924 15:27:39.303435 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.303444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.303482 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.303848 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.303849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.303851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:39.303888 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.303889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.305076 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.305077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.305938 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.305940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.306942 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.306943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.307004 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.307287 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.307337 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.308090 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260924 15:27:39.308298 6485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962de5e0a0e94fe2b01bfe89c131a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:39.311383 6569 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Bootstrap starting.
I20260924 15:27:39.311352 6350 tablet_service.cc:1467] Processing CreateTablet for tablet 962de5e0a0e94fe2b01bfe89c131a2d9 (DEFAULT_TABLE table=test-table [id=a64bd8399d6244baa11c9664a6f4e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:39.311637 6568 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Bootstrap starting.
I20260924 15:27:39.311756 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962de5e0a0e94fe2b01bfe89c131a2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:39.312601 6568 tablet_bootstrap.cc:654] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:39.313190 6568 log.cc:824] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:39.313937 6568 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: No bootstrap required, opened a new log
I20260924 15:27:39.314024 6568 ts_tablet_manager.cc:1397] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:27:39.314057 6569 tablet_bootstrap.cc:654] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:39.314647 6569 log.cc:824] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:39.315387 6572 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Bootstrap starting.
I20260924 15:27:39.315469 6569 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: No bootstrap required, opened a new log
I20260924 15:27:39.315548 6569 ts_tablet_manager.cc:1397] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 15:27:39.316121 6572 tablet_bootstrap.cc:654] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:39.315975 6568 raft_consensus.cc:348] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.316205 6568 raft_consensus.cc:374] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:39.316263 6568 raft_consensus.cc:729] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6364e7dce71649d8b07ee411dfb795b8, State: Initialized, Role: FOLLOWER
I20260924 15:27:39.316392 6568 consensus_queue.cc:260] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.316723 6568 ts_tablet_manager.cc:1428] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:27:39.316711 6572 log.cc:824] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:39.316851 6281 heartbeater.cc:503] Master 127.5.98.254:40361 was elected leader, sending a full tablet report...
I20260924 15:27:39.317457 6572 tablet_bootstrap.cc:492] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: No bootstrap required, opened a new log
I20260924 15:27:39.317569 6572 ts_tablet_manager.cc:1397] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:39.317750 6569 raft_consensus.cc:348] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.317932 6569 raft_consensus.cc:374] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:39.317967 6569 raft_consensus.cc:729] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b756b0d72b6a4a4fb48a9f1585a8c2d6, State: Initialized, Role: FOLLOWER
I20260924 15:27:39.318081 6569 consensus_queue.cc:260] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.318348 6569 ts_tablet_manager.cc:1428] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:39.318353 6551 heartbeater.cc:503] Master 127.5.98.254:40361 was elected leader, sending a full tablet report...
I20260924 15:27:39.319636 6572 raft_consensus.cc:348] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.319842 6572 raft_consensus.cc:374] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:39.319866 6572 raft_consensus.cc:729] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c28418f444406a91bc7e3029461d5, State: Initialized, Role: FOLLOWER
I20260924 15:27:39.319983 6572 consensus_queue.cc:260] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.320255 6572 ts_tablet_manager.cc:1428] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 15:27:39.320255 6416 heartbeater.cc:503] Master 127.5.98.254:40361 was elected leader, sending a full tablet report...
W20260924 15:27:39.353638 6417 tablet.cc:2369] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:39.451413 6282 tablet.cc:2369] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:39.485739 6575 raft_consensus.cc:482] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:39.485868 6575 raft_consensus.cc:504] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.486308 6575 leader_election.cc:288] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b66c28418f444406a91bc7e3029461d5 (127.5.98.194:46145), 6364e7dce71649d8b07ee411dfb795b8 (127.5.98.193:44779)
I20260924 15:27:39.489606 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.486530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44779 (local address 127.5.98.195:41443)
0924 15:27:39.486600 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.486611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.486665 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.486829 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.486831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.486833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:39.486913 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.486915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.487896 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.487897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.488636 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.488638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.489330 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.489331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.489412 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.489416 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.489493 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.489532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 15:27:39.489614 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.486586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:41443 (local address 127.5.98.193:44779)
0924 15:27:39.486635 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.486637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.486656 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:27:39.486723 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.486724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.486733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.486822 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:39.487010 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.487816 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.488733 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.489256 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.489315 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.489322 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.489506 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.489561 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:27:39.490046 6235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962de5e0a0e94fe2b01bfe89c131a2d9" candidate_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6364e7dce71649d8b07ee411dfb795b8" is_pre_election: true
I20260924 15:27:39.490154 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.486530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46145 (local address 127.5.98.195:58481)
0924 15:27:39.486871 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:39.486885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:39.486954 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:39.487156 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:39.487157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:39.487159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:39.487231 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.487232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.488105 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.488106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.489057 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:39.489059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.489858 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.489859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:39.489927 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.489942 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:27:39.490001 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:27:39.490082 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":182,"thread_start_us":103,"threads_started":1}
I20260924 15:27:39.490146 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.486606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:58481 (local address 127.5.98.194:46145)
0924 15:27:39.486662 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.486664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.486951 (+ 287us) server_negotiation.cc:407] Connection header received
0924 15:27:39.487069 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.487070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.487082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.487126 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:39.487340 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.488023 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.489145 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.489694 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.489766 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.489774 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.490023 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.490078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:27:39.490232 6235 raft_consensus.cc:2454] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b756b0d72b6a4a4fb48a9f1585a8c2d6 in term 0.
I20260924 15:27:39.490497 6437 leader_election.cc:302] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6364e7dce71649d8b07ee411dfb795b8, b756b0d72b6a4a4fb48a9f1585a8c2d6; no voters:
I20260924 15:27:39.490473 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962de5e0a0e94fe2b01bfe89c131a2d9" candidate_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c28418f444406a91bc7e3029461d5" is_pre_election: true
I20260924 15:27:39.490619 6370 raft_consensus.cc:2454] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b756b0d72b6a4a4fb48a9f1585a8c2d6 in term 0.
I20260924 15:27:39.490629 6575 raft_consensus.cc:2781] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:39.490695 6575 raft_consensus.cc:482] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:39.490729 6575 raft_consensus.cc:3037] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:39.491416 6575 raft_consensus.cc:504] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.491566 6575 leader_election.cc:288] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [CANDIDATE]: Term 1 election: Requested vote from peers b66c28418f444406a91bc7e3029461d5 (127.5.98.194:46145), 6364e7dce71649d8b07ee411dfb795b8 (127.5.98.193:44779)
I20260924 15:27:39.491703 6370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962de5e0a0e94fe2b01bfe89c131a2d9" candidate_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b66c28418f444406a91bc7e3029461d5"
I20260924 15:27:39.491748 6235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "962de5e0a0e94fe2b01bfe89c131a2d9" candidate_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6364e7dce71649d8b07ee411dfb795b8"
I20260924 15:27:39.491773 6370 raft_consensus.cc:3037] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:39.491835 6235 raft_consensus.cc:3037] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:39.492532 6370 raft_consensus.cc:2454] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b66c28418f444406a91bc7e3029461d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b756b0d72b6a4a4fb48a9f1585a8c2d6 in term 1.
I20260924 15:27:39.492552 6235 raft_consensus.cc:2454] T 962de5e0a0e94fe2b01bfe89c131a2d9 P 6364e7dce71649d8b07ee411dfb795b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b756b0d72b6a4a4fb48a9f1585a8c2d6 in term 1.
I20260924 15:27:39.492710 6440 leader_election.cc:302] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c28418f444406a91bc7e3029461d5, b756b0d72b6a4a4fb48a9f1585a8c2d6; no voters:
I20260924 15:27:39.492833 6575 raft_consensus.cc:2781] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:39.493021 6575 raft_consensus.cc:686] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [term 1 LEADER]: Becoming Leader. State: Replica: b756b0d72b6a4a4fb48a9f1585a8c2d6, State: Running, Role: LEADER
I20260924 15:27:39.493115 6575 consensus_queue.cc:237] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } }
I20260924 15:27:39.493989 6087 catalog_manager.cc:5495] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b756b0d72b6a4a4fb48a9f1585a8c2d6 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b756b0d72b6a4a4fb48a9f1585a8c2d6" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 38311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b66c28418f444406a91bc7e3029461d5" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6364e7dce71649d8b07ee411dfb795b8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44779 } health_report { overall_health: UNKNOWN } } }
W20260924 15:27:39.526932 6552 tablet.cc:2369] T 962de5e0a0e94fe2b01bfe89c131a2d9 P b756b0d72b6a4a4fb48a9f1585a8c2d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:39.541381 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.533431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.5.98.193:44779)
0924 15:27:39.533529 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.533532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.533653 (+ 121us) server_negotiation.cc:407] Connection header received
0924 15:27:39.533755 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.533756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.533768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.533843 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.534475 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.534962 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.535766 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.535959 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.536018 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.538479 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.538489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.538490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.538496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.540288 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.540635 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.540638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.540638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.540697 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.540868 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.540868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.540869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.540994 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.541108 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.541233 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.541295 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:39.563812 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.557306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.5.98.193:44779)
0924 15:27:39.557381 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.557383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.557421 (+ 38us) server_negotiation.cc:407] Connection header received
0924 15:27:39.557495 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.557496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.557505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.557557 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.558080 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.558522 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.559233 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.559404 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.559451 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.561559 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.561560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.561560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.561566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.562700 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.563058 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.563059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.563059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.563104 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.563268 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.563269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.563269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.563352 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.563418 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.563556 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.563613 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:39.585412 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:39.578645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.5.98.193:44779)
0924 15:27:39.578724 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:39.578726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:39.578737 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:39.578824 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:39.578825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:39.578834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:39.578882 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:39.579390 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.579926 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.580738 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:39.580927 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:39.581029 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:39.583203 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:39.583204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:39.583204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:39.583210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:39.584584 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.584868 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.584869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.584869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.584925 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:39.585074 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:39.585074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:39.585075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:39.585152 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:39.585216 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:39.585295 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:27:39.585344 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 15:27:39.585884 6195 tablet_service.cc:2908] Rejecting scan request for tablet 962de5e0a0e94fe2b01bfe89c131a2d9: Uninitialized: clean time has not yet been initialized
I20260924 15:27:39.586844 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6151
I20260924 15:27:39.592000 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6286
I20260924 15:27:39.596372 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6421
I20260924 15:27:39.600657 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6056
2026-09-24T15:27:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2137 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6592](info): set up 2 sockets
Sep 24 15:27:39 dist-test-slave-4g6f krb5kdc[6592](info): commencing operation
krb5kdc: starting...
W20260924 15:27:40.652227 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:40 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:40.780426 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43347
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:40.859491 6605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:40.859750 6605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:40.859790 6605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:40.860917 6605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:40.860986 6605 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:27:40.861008 6605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:40.861030 6605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:40.861059 6605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:40.861081 6605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:40.862519 6605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:40.862777 6605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:40.863001 6605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:40.867923 6610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:40.868006 6611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:40.867931 6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:40.867947 6605 server_base.cc:1034] running on GCE node
I20260924 15:27:40.868319 6605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:40.868594 6605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:40.869773 6605 hybrid_clock.cc:648] HybridClock initialized: now 1790263660869768 us; error 36 us; skew 500 ppm
Sep 24 15:27:40 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:40.872351 6605 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:40.872723 6605 webserver.cc:460] Webserver started at http://127.5.98.254:46419/ using document root <none> and password file <none>
I20260924 15:27:40.872889 6605 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:40.872927 6605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:40.873018 6605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:40.873967 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "07d020f2c90a4d238e82891e9c0ab9e3"
format_stamp: "Formatted at 2026-09-24 15:27:40 on dist-test-slave-4g6f"
server_key: "95ce177b5c5157ae5c1cc84df08c2a21"
server_key_iv: "12b9e2ef797dfa5b26df4b7fea0bcac3"
server_key_version: "encryptionkey@0"
I20260924 15:27:40.874316 6605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "07d020f2c90a4d238e82891e9c0ab9e3"
format_stamp: "Formatted at 2026-09-24 15:27:40 on dist-test-slave-4g6f"
server_key: "95ce177b5c5157ae5c1cc84df08c2a21"
server_key_iv: "12b9e2ef797dfa5b26df4b7fea0bcac3"
server_key_version: "encryptionkey@0"
I20260924 15:27:40.875638 6605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:40.876502 6620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:40.876680 6605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:40.876771 6605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "07d020f2c90a4d238e82891e9c0ab9e3"
format_stamp: "Formatted at 2026-09-24 15:27:40 on dist-test-slave-4g6f"
server_key: "95ce177b5c5157ae5c1cc84df08c2a21"
server_key_iv: "12b9e2ef797dfa5b26df4b7fea0bcac3"
server_key_version: "encryptionkey@0"
I20260924 15:27:40.876828 6605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:40.885329 6605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:40.887531 6605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:40.887696 6605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:40.892042 6605 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43347
I20260924 15:27:40.892138 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43347 every 8 connection(s)
I20260924 15:27:40.892439 6605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:40.893114 6684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:40.894510 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6605
I20260924 15:27:40.894606 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:40.894767 5515 external_mini_cluster.cc:1428] Setting key bfe43d51767b7d847636e267daa6000b
I20260924 15:27:40.895835 6684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Bootstrap starting.
I20260924 15:27:40.896519 6684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:40.896798 6684 log.cc:824] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:40.897459 6684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: No bootstrap required, opened a new log
I20260924 15:27:40.899202 6684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } }
I20260924 15:27:40.899359 6684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:40.899407 6684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d020f2c90a4d238e82891e9c0ab9e3, State: Initialized, Role: FOLLOWER
I20260924 15:27:40.899535 6684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } }
I20260924 15:27:40.899626 6684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:40.899712 6684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:40.899770 6684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:40.900267 6684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } }
I20260924 15:27:40.900381 6684 leader_election.cc:302] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07d020f2c90a4d238e82891e9c0ab9e3; no voters:
I20260924 15:27:40.900544 6684 leader_election.cc:288] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:40.900584 6689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:40.900699 6689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 07d020f2c90a4d238e82891e9c0ab9e3, State: Running, Role: LEADER
I20260924 15:27:40.900799 6684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:40.900835 6689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } }
I20260924 15:27:40.901189 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } } }
Sep 24 15:27:40 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:40.901338 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:40.901551 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07d020f2c90a4d238e82891e9c0ab9e3. Latest consensus state: current_term: 1 leader_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07d020f2c90a4d238e82891e9c0ab9e3" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43347 } } }
I20260924 15:27:40.901659 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:40.901821 6698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:40.902390 6698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:40.904845 6698 catalog_manager.cc:1347] Generated new cluster ID: f9170dd77ec24b4d9d42d345a7db5486
I20260924 15:27:40.905031 6698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:40.906613 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:40.895271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.5.98.254:43347)
0924 15:27:40.895468 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:40.895477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:40.895507 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:27:40.896052 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:40.896061 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:40.896118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:40.896285 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:40.896987 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:40.897968 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:40.898702 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:40.899120 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:40.899235 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:40.901921 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:40.901925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:40.901926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:40.901950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:40.904897 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:40.905311 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:40.905314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:40.905314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:40.905414 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:40.905613 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:40.905613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:40.905614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:40.905882 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:40.906017 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:40.906257 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:27:40.906398 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260924 15:27:40.936550 6698 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:40.937312 6698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:40.946319 6698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Generated new TSK 0
I20260924 15:27:40.946511 6698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:40.976209 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43347
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:41.057159 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:41.057354 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:41.057379 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:41.058521 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:41.058561 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:41.058583 6710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:41.058599 6710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:41.060036 6710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:41.060334 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:41.060560 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:41.065544 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.065587 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.065567 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:41.065785 6710 server_base.cc:1034] running on GCE node
I20260924 15:27:41.065989 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:41.066275 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:41.067423 6710 hybrid_clock.cc:648] HybridClock initialized: now 1790263661067402 us; error 36 us; skew 500 ppm
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:41.069984 6710 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:41.070447 6710 webserver.cc:460] Webserver started at http://127.5.98.193:40719/ using document root <none> and password file <none>
I20260924 15:27:41.070744 6710 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:41.070812 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:41.070967 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:41.072062 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "99f6918bf7d24c5ab5b0b32f15db63c6"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "aedcaa73493320cb97d4359f85719885"
server_key_iv: "3632994b2f61c9f918fbb8c2af751cfe"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.072422 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "99f6918bf7d24c5ab5b0b32f15db63c6"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "aedcaa73493320cb97d4359f85719885"
server_key_iv: "3632994b2f61c9f918fbb8c2af751cfe"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.073781 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:41.074514 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.074678 6710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:41.074779 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "99f6918bf7d24c5ab5b0b32f15db63c6"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "aedcaa73493320cb97d4359f85719885"
server_key_iv: "3632994b2f61c9f918fbb8c2af751cfe"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.074851 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:41.096692 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:41.098515 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:41.098626 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:41.099048 6710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:41.099080 6710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.099102 6710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:41.099143 6710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.105814 6710 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:37127
I20260924 15:27:41.106168 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:41.106456 6841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:37127 every 8 connection(s)
I20260924 15:27:41.109444 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.099686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:52799 (local address 127.5.98.254:43347)
0924 15:27:41.099749 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.099752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.099965 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:27:41.100436 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.100438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.100451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.100544 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.101376 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.102280 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.103867 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.104207 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.104281 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.106741 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.106743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.106743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.106751 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.108054 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.108451 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.108452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.108452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.108513 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.108772 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.108773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.108774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.108885 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.108964 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.109233 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.109351 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:27:41.109622 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.099696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.193:52799)
0924 15:27:41.099863 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.099896 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.100338 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.100643 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.100647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.100974 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.101264 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.101270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.102382 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.102384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.103763 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.103766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.104297 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.104298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.104400 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.104956 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.104966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.106644 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.108150 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.108157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.108159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.108365 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.108587 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.108587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.108588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.108699 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.108960 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.108968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.109110 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.109212 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.109386 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260924 15:27:41.110093 6842 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43347
I20260924 15:27:41.110195 6842 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:41.110361 6842 heartbeater.cc:511] Master 127.5.98.254:43347 requested a full tablet report, sending...
I20260924 15:27:41.110845 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6710
I20260924 15:27:41.110895 6637 ts_manager.cc:194] Registered new tserver with Master: 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193:37127)
I20260924 15:27:41.110945 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:41.111121 5515 external_mini_cluster.cc:1428] Setting key 84f6805963190ae1bdfe1fb5af5bb2af
I20260924 15:27:41.111832 6637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:52799
I20260924 15:27:41.117677 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.111097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.193:33109)
0924 15:27:41.111291 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.111304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.111373 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.111519 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.111519 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.111741 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.111812 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.111814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.112695 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.112696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.113609 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.113612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.114124 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.114125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.114190 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.114588 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.114592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.114872 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.116735 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.116736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.116737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.116893 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.117108 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.117109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.117109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.117157 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.117442 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.117443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.117487 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.117538 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.117610 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":138,"threads_started":1}
I20260924 15:27:41.117796 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.111166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:33109 (local address 127.5.98.254:43347)
0924 15:27:41.111203 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.111205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.111377 (+ 172us) server_negotiation.cc:407] Connection header received
0924 15:27:41.111449 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.111450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.111460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.111515 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.111888 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.112555 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.113709 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.114032 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.114116 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.114957 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.114959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.114959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.114967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.116639 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.116972 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.116974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.116974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.117036 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.117221 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.117222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.117223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.117361 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.117450 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.117591 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.117672 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:41.139150 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43347
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:41.220597 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:41.220777 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:41.220798 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:41.221916 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:41.221954 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:41.221974 6851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:41.221992 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:41.223301 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:41.223472 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:41.223629 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:41.228435 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.228502 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.228442 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:41.228729 6851 server_base.cc:1034] running on GCE node
I20260924 15:27:41.229015 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:41.229262 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:41.230427 6851 hybrid_clock.cc:648] HybridClock initialized: now 1790263661230409 us; error 33 us; skew 500 ppm
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:41.232980 6851 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:41.233481 6851 webserver.cc:460] Webserver started at http://127.5.98.194:45443/ using document root <none> and password file <none>
I20260924 15:27:41.233704 6851 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:41.233771 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:41.233932 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:41.234961 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "bf55583854d1428ba5bfe6d840d437f9"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5693827a393e1c3347eaea4784d7d57c"
server_key_iv: "0991a9ce688f3e044ec1f7f595646972"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.235342 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "bf55583854d1428ba5bfe6d840d437f9"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5693827a393e1c3347eaea4784d7d57c"
server_key_iv: "0991a9ce688f3e044ec1f7f595646972"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.236835 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:41.237691 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.237901 6851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:41.238009 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "bf55583854d1428ba5bfe6d840d437f9"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5693827a393e1c3347eaea4784d7d57c"
server_key_iv: "0991a9ce688f3e044ec1f7f595646972"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.238087 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:41.247836 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:41.249686 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:41.249848 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:41.250330 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:41.250394 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.250435 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:41.250481 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.256999 6851 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:45095
I20260924 15:27:41.257082 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:45095 every 8 connection(s)
I20260924 15:27:41.257355 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:41.260540 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.250863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:51999 (local address 127.5.98.254:43347)
0924 15:27:41.250915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.250917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.251154 (+ 237us) server_negotiation.cc:407] Connection header received
0924 15:27:41.251725 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.251727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.251739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.251795 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.252628 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.253506 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.255068 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.255354 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.255451 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.258068 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.258069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.258069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.258077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.259251 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.259638 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.259639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.259639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.259737 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.260007 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.260008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.260008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.260123 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.260176 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.260407 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.260468 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:27:41.260787 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.250854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.194:51999)
0924 15:27:41.251037 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.251071 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.251592 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.251865 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.251869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.252217 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.252511 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.252519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.253610 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.253611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.254993 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.254996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.255461 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.255462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.255611 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.256194 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.256204 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.257981 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.259322 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.259328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.259330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.259570 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.259830 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.259831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.259832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.259940 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.260184 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.260189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.260299 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.260382 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.260524 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260924 15:27:41.261283 6983 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43347
I20260924 15:27:41.261384 6983 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:41.261557 6983 heartbeater.cc:511] Master 127.5.98.254:43347 requested a full tablet report, sending...
I20260924 15:27:41.261929 6638 ts_manager.cc:194] Registered new tserver with Master: bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095)
I20260924 15:27:41.262408 6638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:51999
I20260924 15:27:41.263731 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6851
I20260924 15:27:41.263821 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:41.263990 5515 external_mini_cluster.cc:1428] Setting key 7cb9a850131436196dc0c06daefdff56
I20260924 15:27:41.268257 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.262381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:60771 (local address 127.5.98.254:43347)
0924 15:27:41.262428 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.262430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.262484 (+ 54us) server_negotiation.cc:407] Connection header received
0924 15:27:41.262582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.262583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.262594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.262654 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.262981 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.263451 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.264462 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.264715 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.264812 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.265676 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.265677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.265678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.265685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.267065 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.267425 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.267426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.267426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.267480 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.267711 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.267712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.267712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.267856 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.267936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.268093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.268169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:27:41.268270 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.262301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.194:60771)
0924 15:27:41.262430 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.262442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.262499 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.262669 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.262669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.262828 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.262876 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.262877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.263543 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.263544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.264358 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.264378 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.264816 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.264817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.264917 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.265333 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.265337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.265590 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.267168 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.267169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.267170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.267345 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.267556 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.267557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.267557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.267614 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.267930 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.267931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.267980 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.268069 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.268168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:41.288316 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43347
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:41.374177 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:41.374370 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:41.374393 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:41.375552 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:41.375593 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:41.375614 6992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:41.375633 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:41.377123 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:41.377351 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:41.377614 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:41.382941 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.382947 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:41.383040 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:41.383220 6992 server_base.cc:1034] running on GCE node
I20260924 15:27:41.383426 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:41.383622 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:41.384802 6992 hybrid_clock.cc:648] HybridClock initialized: now 1790263661384789 us; error 31 us; skew 500 ppm
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:41.387153 6992 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:41.387524 6992 webserver.cc:460] Webserver started at http://127.5.98.195:35841/ using document root <none> and password file <none>
I20260924 15:27:41.387804 6992 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:41.387849 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:41.387940 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:41.388931 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "d5b8e06a829a4793881ab01990d868c4"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5d031cd438cb5f5fec01ba949720eb36"
server_key_iv: "351af70ac3c86e15c93d575c30f88c84"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.389231 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "d5b8e06a829a4793881ab01990d868c4"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5d031cd438cb5f5fec01ba949720eb36"
server_key_iv: "351af70ac3c86e15c93d575c30f88c84"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.390513 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:41.391263 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.391472 6992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:41.391607 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "d5b8e06a829a4793881ab01990d868c4"
format_stamp: "Formatted at 2026-09-24 15:27:41 on dist-test-slave-4g6f"
server_key: "5d031cd438cb5f5fec01ba949720eb36"
server_key_iv: "351af70ac3c86e15c93d575c30f88c84"
server_key_version: "encryptionkey@0"
I20260924 15:27:41.391714 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:41.407292 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:41.409247 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:41.409403 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:41.409888 6992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:41.409951 6992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.410012 6992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:41.410058 6992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:41.417297 6992 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:37519
I20260924 15:27:41.417382 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:37519 every 8 connection(s)
I20260924 15:27:41.417696 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:41.419790 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.410485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:36913 (local address 127.5.98.254:43347)
0924 15:27:41.410558 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.410562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.410777 (+ 215us) server_negotiation.cc:407] Connection header received
0924 15:27:41.411242 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.411244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.411258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.411360 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.412124 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.412626 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.413766 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.414028 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.414112 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.417121 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.417123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.417124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.417131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.418646 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.418991 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.418992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.418992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.419043 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.419219 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.419220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.419220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.419319 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.419368 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.419608 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.419720 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:41.419956 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.410414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.195:36913)
0924 15:27:41.410674 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.410703 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.411150 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.411507 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.411511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.411834 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.412022 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.412028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.412731 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.412732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.413659 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.413661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.414122 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.414123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.414252 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.414709 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.414718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.416993 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.418717 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.418720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.418722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.418922 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.419088 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.419088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.419088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.419167 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.419376 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.419381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.419488 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.419570 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.419743 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":75,"threads_started":1}
I20260924 15:27:41.420408 7124 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43347
I20260924 15:27:41.420478 7124 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:41.420675 7124 heartbeater.cc:511] Master 127.5.98.254:43347 requested a full tablet report, sending...
I20260924 15:27:41.421031 6637 ts_manager.cc:194] Registered new tserver with Master: d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519)
I20260924 15:27:41.421443 6637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:36913
I20260924 15:27:41.423128 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 6992
I20260924 15:27:41.423215 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:41.423365 5515 external_mini_cluster.cc:1428] Setting key 772936fe12e17575c62b90bebd0ac11c
I20260924 15:27:41.424258 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:41.427466 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.421506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.5.98.195:43985)
0924 15:27:41.421692 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.421705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.421785 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.421898 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.421899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.422092 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.422161 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.422163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.422761 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.422762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.423497 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.423500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.424000 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.424001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.424075 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.424514 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.424519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.424820 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.426406 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.426407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.426407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.426675 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.426853 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.426854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.426854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.426929 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.427139 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.427140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.427195 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.427267 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.427374 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":3,"thread_start_us":137,"threads_started":1}
I20260924 15:27:41.427537 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.421581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:43985 (local address 127.5.98.254:43347)
0924 15:27:41.421618 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.421620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.421771 (+ 151us) server_negotiation.cc:407] Connection header received
0924 15:27:41.421841 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.421842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.421849 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.421892 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.422233 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.422682 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.423623 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.423904 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.423985 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.424902 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.424903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.424904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.424910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.426327 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.426750 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.426750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.426751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.426796 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.426986 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.426987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.426987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.427077 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.427125 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.427312 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.427386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:41.447033 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.440085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.5.98.254:43347)
0924 15:27:41.440145 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.440147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.440155 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:41.440178 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.440179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.440191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.440247 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.440895 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.441359 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.442109 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.442284 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.442336 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.444556 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.444557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.444557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.444563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.445862 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.446265 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.446266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.446267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.446323 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.446527 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.446528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.446529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.446632 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.446737 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.446862 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.446932 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:27:41.448935 6637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:41.450423 6637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:41.456671 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.452684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:37127 (local address 127.0.0.1:52200)
0924 15:27:41.452995 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.453006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.453062 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.453475 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.453476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.453478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.453527 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.453529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454376 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.454377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.455536 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.455538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.456360 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.456437 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.456453 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.456533 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.456586 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":29,"thread_start_us":82,"threads_started":1}
I20260924 15:27:41.456684 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.452784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.5.98.193:37127)
0924 15:27:41.453312 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.453317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.453327 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:41.453364 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.453368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.453387 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.453469 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.453614 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454291 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.455634 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.456275 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456350 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.456410 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.456554 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.456593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":86,"threads_started":1}
I20260924 15:27:41.457049 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.452672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:37519 (local address 127.0.0.1:60406)
0924 15:27:41.452823 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.452865 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.452928 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.453185 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.453189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.453190 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.453260 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.453263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454603 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.454604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.455664 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.455689 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.456808 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.456872 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.456876 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.456934 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.456979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":51,"threads_started":1}
I20260924 15:27:41.457347 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.452874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:45095 (local address 127.0.0.1:35034)
0924 15:27:41.453080 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.453090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.453136 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.453754 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.453755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.453757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.453801 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.453802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454999 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.455000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.456085 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.456088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.457070 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.457071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.457205 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.457209 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.457254 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.457290 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 15:27:41.457399 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.453067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.5.98.194:45095)
0924 15:27:41.453267 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.453272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.453301 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:27:41.453340 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.453577 (+ 237us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.453605 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.453682 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.453873 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454909 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456174 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.456931 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456989 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.457038 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.457270 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.457304 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260924 15:27:41.457383 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.452841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.5.98.195:37519)
0924 15:27:41.453009 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.453015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.453027 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:27:41.453069 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.453072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.453089 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.453173 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.453378 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.454501 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.455750 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.456477 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.456541 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.456594 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.457252 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.457293 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260924 15:27:41.457518 6776 tablet_service.cc:1467] Processing CreateTablet for tablet 70acb4042a3c49a29682c44646a83695 (DEFAULT_TABLE table=test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.457896 6776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70acb4042a3c49a29682c44646a83695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.457998 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 70acb4042a3c49a29682c44646a83695 (DEFAULT_TABLE table=test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.458329 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70acb4042a3c49a29682c44646a83695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.460109 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 70acb4042a3c49a29682c44646a83695 (DEFAULT_TABLE table=test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.460415 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70acb4042a3c49a29682c44646a83695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.460938 7148 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Bootstrap starting.
I20260924 15:27:41.461542 7149 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Bootstrap starting.
I20260924 15:27:41.461802 7148 tablet_bootstrap.cc:654] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.462142 7148 log.cc:824] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:41.462167 7149 tablet_bootstrap.cc:654] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.462577 7149 log.cc:824] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:41.462904 7148 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: No bootstrap required, opened a new log
I20260924 15:27:41.462988 7148 ts_tablet_manager.cc:1397] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:41.463299 7149 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: No bootstrap required, opened a new log
I20260924 15:27:41.463378 7149 ts_tablet_manager.cc:1397] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:41.463760 7152 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Bootstrap starting.
I20260924 15:27:41.464666 7152 tablet_bootstrap.cc:654] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.465018 7152 log.cc:824] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:41.465116 7148 raft_consensus.cc:348] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.465279 7148 raft_consensus.cc:374] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.465324 7148 raft_consensus.cc:729] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6918bf7d24c5ab5b0b32f15db63c6, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.465305 7149 raft_consensus.cc:348] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.465472 7149 raft_consensus.cc:374] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.465435 7148 consensus_queue.cc:260] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.465512 7149 raft_consensus.cc:729] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf55583854d1428ba5bfe6d840d437f9, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.465636 7149 consensus_queue.cc:260] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.465706 7148 ts_tablet_manager.cc:1428] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:41.465835 6842 heartbeater.cc:503] Master 127.5.98.254:43347 was elected leader, sending a full tablet report...
I20260924 15:27:41.465893 7149 ts_tablet_manager.cc:1428] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:27:41.466010 6983 heartbeater.cc:503] Master 127.5.98.254:43347 was elected leader, sending a full tablet report...
I20260924 15:27:41.466291 7152 tablet_bootstrap.cc:492] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: No bootstrap required, opened a new log
I20260924 15:27:41.466364 7152 ts_tablet_manager.cc:1397] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:27:41.468000 7152 raft_consensus.cc:348] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.468117 7152 raft_consensus.cc:374] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.468138 7152 raft_consensus.cc:729] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b8e06a829a4793881ab01990d868c4, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.468268 7152 consensus_queue.cc:260] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.468508 7152 ts_tablet_manager.cc:1428] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:41.468550 7124 heartbeater.cc:503] Master 127.5.98.254:43347 was elected leader, sending a full tablet report...
W20260924 15:27:41.508025 6984 tablet.cc:2369] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:41.519977 7156 raft_consensus.cc:482] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:41.520133 7156 raft_consensus.cc:504] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.520563 7156 leader_election.cc:288] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193:37127), bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095)
I20260924 15:27:41.524422 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.520803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:51811 (local address 127.5.98.193:37127)
0924 15:27:41.520858 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.520860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.521191 (+ 331us) server_negotiation.cc:407] Connection header received
0924 15:27:41.521254 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.521255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.521267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.521330 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.521455 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.522168 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.523436 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.524031 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.524105 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.524114 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.524304 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.524351 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:27:41.524423 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.520725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:45095 (local address 127.5.98.195:52681)
0924 15:27:41.520778 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.520791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.520835 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.521014 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.521015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.521017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.521127 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.521129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.522097 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.522098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.523326 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.523328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.524118 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.524119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.524210 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.524217 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.524278 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.524329 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":11}
I20260924 15:27:41.524422 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.520826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:52681 (local address 127.5.98.194:45095)
0924 15:27:41.520880 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.520882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.520891 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:41.520918 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.520919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.520933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.520990 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.521202 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.521997 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.523436 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.524025 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.524105 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.524114 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.524304 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.524351 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:27:41.524430 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.520731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:37127 (local address 127.5.98.195:51811)
0924 15:27:41.521121 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.521133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.521200 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.521332 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.521333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.521334 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.521391 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.521392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.522257 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.522258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.523329 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.523330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.524118 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.524119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.524210 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.524217 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.524278 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.524329 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":158,"thread_start_us":140,"threads_started":1}
I20260924 15:27:41.524755 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70acb4042a3c49a29682c44646a83695" candidate_uuid: "d5b8e06a829a4793881ab01990d868c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9" is_pre_election: true
I20260924 15:27:41.524834 6796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70acb4042a3c49a29682c44646a83695" candidate_uuid: "d5b8e06a829a4793881ab01990d868c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" is_pre_election: true
I20260924 15:27:41.524899 6937 raft_consensus.cc:2454] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b8e06a829a4793881ab01990d868c4 in term 0.
I20260924 15:27:41.525031 6796 raft_consensus.cc:2454] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b8e06a829a4793881ab01990d868c4 in term 0.
I20260924 15:27:41.525189 7011 leader_election.cc:302] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf55583854d1428ba5bfe6d840d437f9, d5b8e06a829a4793881ab01990d868c4; no voters:
I20260924 15:27:41.525372 7156 raft_consensus.cc:2781] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:41.525429 7156 raft_consensus.cc:482] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:41.525456 7156 raft_consensus.cc:3037] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.526154 7156 raft_consensus.cc:504] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.526300 7156 leader_election.cc:288] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193:37127), bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095)
I20260924 15:27:41.526437 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70acb4042a3c49a29682c44646a83695" candidate_uuid: "d5b8e06a829a4793881ab01990d868c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9"
I20260924 15:27:41.526500 6796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70acb4042a3c49a29682c44646a83695" candidate_uuid: "d5b8e06a829a4793881ab01990d868c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6"
I20260924 15:27:41.526518 6937 raft_consensus.cc:3037] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.526592 6796 raft_consensus.cc:3037] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.527231 6937 raft_consensus.cc:2454] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b8e06a829a4793881ab01990d868c4 in term 1.
I20260924 15:27:41.527295 6796 raft_consensus.cc:2454] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b8e06a829a4793881ab01990d868c4 in term 1.
I20260924 15:27:41.527418 7011 leader_election.cc:302] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf55583854d1428ba5bfe6d840d437f9, d5b8e06a829a4793881ab01990d868c4; no voters:
I20260924 15:27:41.527576 7156 raft_consensus.cc:2781] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:41.527765 7156 raft_consensus.cc:686] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 1 LEADER]: Becoming Leader. State: Replica: d5b8e06a829a4793881ab01990d868c4, State: Running, Role: LEADER
I20260924 15:27:41.527863 7156 consensus_queue.cc:237] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } }
I20260924 15:27:41.528669 6638 catalog_manager.cc:5495] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b8e06a829a4793881ab01990d868c4 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "d5b8e06a829a4793881ab01990d868c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } health_report { overall_health: UNKNOWN } } }
I20260924 15:27:41.536345 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.533787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.5.98.195:37519)
0924 15:27:41.533857 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.533859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.533866 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:41.533938 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.533939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.533953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.533998 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:41.534115 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.534559 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.535327 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.535531 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.535593 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.535820 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.535929 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.536004 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.536048 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.536066 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:27:41.537957 6937 raft_consensus.cc:1261] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Refusing update from remote peer d5b8e06a829a4793881ab01990d868c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.537958 6796 raft_consensus.cc:1261] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Refusing update from remote peer d5b8e06a829a4793881ab01990d868c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.538301 7158 consensus_queue.cc:1035] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.538401 7156 consensus_queue.cc:1035] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.540630 7161 mvcc.cc:205] Tried to move back new op lower bound from 7332919957657059328 to 7332919957618900992. Current Snapshot: MvccSnapshot[applied={T|T < 7332919957657059328}]
I20260924 15:27:41.541386 7162 mvcc.cc:205] Tried to move back new op lower bound from 7332919957657059328 to 7332919957618900992. Current Snapshot: MvccSnapshot[applied={T|T < 7332919957657059328}]
I20260924 15:27:41.541491 7163 mvcc.cc:205] Tried to move back new op lower bound from 7332919957657059328 to 7332919957618900992. Current Snapshot: MvccSnapshot[applied={T|T < 7332919957657059328 or (T in {7332919957657059328})}]
I20260924 15:27:41.544239 6637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:41.544337 6637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:41.545076 6637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Sending DeleteTablet for 3 replicas of tablet 70acb4042a3c49a29682c44646a83695
I20260924 15:27:41.545393 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 70acb4042a3c49a29682c44646a83695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:60406
I20260924 15:27:41.545404 6776 tablet_service.cc:1514] Processing DeleteTablet for tablet 70acb4042a3c49a29682c44646a83695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:52200
I20260924 15:27:41.545416 6917 tablet_service.cc:1514] Processing DeleteTablet for tablet 70acb4042a3c49a29682c44646a83695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:35034
I20260924 15:27:41.545712 7171 tablet_replica.cc:330] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: stopping tablet replica
I20260924 15:27:41.545850 7171 raft_consensus.cc:2229] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:41.545883 7172 tablet_replica.cc:330] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: stopping tablet replica
I20260924 15:27:41.545886 6637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 15:27:41.545996 7172 raft_consensus.cc:2229] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 15:27:41.546034 6637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:41.546110 7172 raft_consensus.cc:2258] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.546478 7170 tablet_replica.cc:330] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: stopping tablet replica
I20260924 15:27:41.546567 7172 ts_tablet_manager.cc:1905] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:41.546589 7170 raft_consensus.cc:2229] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:41.546674 7170 raft_consensus.cc:2258] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.547101 7171 raft_consensus.cc:2258] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.547508 6776 tablet_service.cc:1467] Processing CreateTablet for tablet 746b619c348e4355ae1a1cd107999e2c (DEFAULT_TABLE table=test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.547621 6776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746b619c348e4355ae1a1cd107999e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.547767 7170 ts_tablet_manager.cc:1905] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:41.548326 7172 ts_tablet_manager.cc:1918] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:41.548390 7172 log.cc:1196] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/70acb4042a3c49a29682c44646a83695
I20260924 15:27:41.548650 7172 ts_tablet_manager.cc:1939] T 70acb4042a3c49a29682c44646a83695 P bf55583854d1428ba5bfe6d840d437f9: Deleting consensus metadata
I20260924 15:27:41.549080 7148 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Bootstrap starting.
I20260924 15:27:41.549448 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 746b619c348e4355ae1a1cd107999e2c (DEFAULT_TABLE table=test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.549443 6625 catalog_manager.cc:4841] TS bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095): tablet 70acb4042a3c49a29682c44646a83695 (table test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]) successfully deleted
I20260924 15:27:41.549554 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746b619c348e4355ae1a1cd107999e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.549580 7148 tablet_bootstrap.cc:654] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.550195 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 746b619c348e4355ae1a1cd107999e2c (DEFAULT_TABLE table=test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:41.550302 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 746b619c348e4355ae1a1cd107999e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.550814 7152 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Bootstrap starting.
I20260924 15:27:41.551222 7152 tablet_bootstrap.cc:654] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.551435 7149 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Bootstrap starting.
I20260924 15:27:41.551721 7148 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: No bootstrap required, opened a new log
I20260924 15:27:41.551776 7148 ts_tablet_manager.cc:1397] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:27:41.551941 7148 raft_consensus.cc:348] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.551996 7149 tablet_bootstrap.cc:654] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.552027 7148 raft_consensus.cc:374] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.552057 7148 raft_consensus.cc:729] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6918bf7d24c5ab5b0b32f15db63c6, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.552002 7171 ts_tablet_manager.cc:1905] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:41.552101 7148 consensus_queue.cc:260] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.552223 7148 ts_tablet_manager.cc:1428] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:41.552533 7149 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: No bootstrap required, opened a new log
I20260924 15:27:41.552573 7149 ts_tablet_manager.cc:1397] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:41.552656 7170 ts_tablet_manager.cc:1918] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:41.552721 7170 log.cc:1196] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/70acb4042a3c49a29682c44646a83695
I20260924 15:27:41.552762 7149 raft_consensus.cc:348] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.552838 7149 raft_consensus.cc:374] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.552865 7149 raft_consensus.cc:729] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf55583854d1428ba5bfe6d840d437f9, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.552902 7149 consensus_queue.cc:260] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.552980 7170 ts_tablet_manager.cc:1939] T 70acb4042a3c49a29682c44646a83695 P d5b8e06a829a4793881ab01990d868c4: Deleting consensus metadata
I20260924 15:27:41.553076 7149 ts_tablet_manager.cc:1428] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:41.553690 6622 catalog_manager.cc:4841] TS d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519): tablet 70acb4042a3c49a29682c44646a83695 (table test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]) successfully deleted
I20260924 15:27:41.554527 7171 ts_tablet_manager.cc:1918] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:41.554605 7171 log.cc:1196] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/70acb4042a3c49a29682c44646a83695
I20260924 15:27:41.554605 7152 tablet_bootstrap.cc:492] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: No bootstrap required, opened a new log
I20260924 15:27:41.554651 7152 ts_tablet_manager.cc:1397] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 15:27:41.554795 7152 raft_consensus.cc:348] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.554862 7171 ts_tablet_manager.cc:1939] T 70acb4042a3c49a29682c44646a83695 P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting consensus metadata
I20260924 15:27:41.554870 7152 raft_consensus.cc:374] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.554899 7152 raft_consensus.cc:729] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b8e06a829a4793881ab01990d868c4, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.554942 7152 consensus_queue.cc:260] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.555114 7152 ts_tablet_manager.cc:1428] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:41.555811 6621 catalog_manager.cc:4841] TS 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193:37127): tablet 70acb4042a3c49a29682c44646a83695 (table test-table [id=9cc789a9b0ed4d7a89ef38815b29f8df]) successfully deleted
W20260924 15:27:41.606899 6843 tablet.cc:2369] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:41.668236 7125 tablet.cc:2369] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:41.734632 7154 raft_consensus.cc:482] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:41.734761 7154 raft_consensus.cc:504] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.735193 7154 leader_election.cc:288] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095), d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519)
I20260924 15:27:41.739409 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.735435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:44259 (local address 127.5.98.195:37519)
0924 15:27:41.735490 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.735492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.735961 (+ 469us) server_negotiation.cc:407] Connection header received
0924 15:27:41.736081 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.736083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.736093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.736127 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.736371 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.737105 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.738381 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.738951 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.739029 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.739039 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.739282 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.739331 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:27:41.739401 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.735609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:50053 (local address 127.5.98.194:45095)
0924 15:27:41.735765 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.735816 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.735827 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:41.735918 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.735920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.735939 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.735970 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:41.736216 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.737106 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.738535 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.739075 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.739153 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.739159 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.739304 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.739343 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260924 15:27:41.739413 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.735344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:45095 (local address 127.5.98.193:50053)
0924 15:27:41.735580 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.735626 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.735826 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.736046 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.736047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.736050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.736122 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.736125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.737226 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.737227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.738452 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.738455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.739163 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.739164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.739236 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.739239 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.739296 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.739322 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":55,"threads_started":1}
I20260924 15:27:41.739540 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.735467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:37519 (local address 127.5.98.193:44259)
0924 15:27:41.735879 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.735890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.735994 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.736220 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.736221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.736223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:41.736290 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.736292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.737206 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.737207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.738273 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.738276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.739031 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.739032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.739156 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.739161 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.739266 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.739313 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":73,"threads_started":1}
I20260924 15:27:41.739724 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746b619c348e4355ae1a1cd107999e2c" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9" is_pre_election: true
I20260924 15:27:41.739821 6937 raft_consensus.cc:2454] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 0.
I20260924 15:27:41.739867 7078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746b619c348e4355ae1a1cd107999e2c" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b8e06a829a4793881ab01990d868c4" is_pre_election: true
I20260924 15:27:41.739998 7078 raft_consensus.cc:2454] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 0.
I20260924 15:27:41.740089 6729 leader_election.cc:302] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6918bf7d24c5ab5b0b32f15db63c6, bf55583854d1428ba5bfe6d840d437f9; no voters:
I20260924 15:27:41.740249 7154 raft_consensus.cc:2781] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:41.740286 7154 raft_consensus.cc:482] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:41.740300 7154 raft_consensus.cc:3037] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.741165 7154 raft_consensus.cc:504] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.741323 7154 leader_election.cc:288] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 election: Requested vote from peers bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095), d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519)
I20260924 15:27:41.741556 7078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746b619c348e4355ae1a1cd107999e2c" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b8e06a829a4793881ab01990d868c4"
I20260924 15:27:41.741555 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "746b619c348e4355ae1a1cd107999e2c" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9"
I20260924 15:27:41.741636 6937 raft_consensus.cc:3037] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.741636 7078 raft_consensus.cc:3037] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.742444 7078 raft_consensus.cc:2454] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 1.
I20260924 15:27:41.742443 6937 raft_consensus.cc:2454] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 1.
I20260924 15:27:41.742627 6727 leader_election.cc:302] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6918bf7d24c5ab5b0b32f15db63c6, d5b8e06a829a4793881ab01990d868c4; no voters:
I20260924 15:27:41.742734 7154 raft_consensus.cc:2781] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:41.742861 7154 raft_consensus.cc:686] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 LEADER]: Becoming Leader. State: Replica: 99f6918bf7d24c5ab5b0b32f15db63c6, State: Running, Role: LEADER
I20260924 15:27:41.742940 7154 consensus_queue.cc:237] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.743628 6638 catalog_manager.cc:5495] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263660, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:41.803519 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.796415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.5.98.254:43347)
0924 15:27:41.796510 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.796512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.796658 (+ 146us) server_negotiation.cc:407] Connection header received
0924 15:27:41.796803 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.796804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.796818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.796875 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.797363 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.797858 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.798548 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.798730 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.798788 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.800968 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.800969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.800970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.800976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.802537 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.802875 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.802876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.802876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.802928 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.803100 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.803101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.803101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.803196 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.803244 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.803362 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.803451 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 15:27:41.803730 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.796369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43347 (local address 127.0.0.1:55274)
0924 15:27:41.796546 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.796558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.796680 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.796904 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.796905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.797215 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:41.797264 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.797266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.797946 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.797947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.798465 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.798466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.798800 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.798801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.798841 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.799345 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:41.799349 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:41.800862 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:41.802608 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.802615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.802618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.802809 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.802986 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:41.802986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:41.802987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:41.803045 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:41.803250 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:41.803256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:41.803295 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.803346 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.803565 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":120,"thread_start_us":85,"threads_started":1}
I20260924 15:27:41.806074 6638 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:55274:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:27:41.806274 6638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:41.807963 6776 tablet_service.cc:1467] Processing CreateTablet for tablet 787bd16518fc450492e7b8973869f4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d8ae28c4ff54c7f849747b3df988e92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:41.808024 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 787bd16518fc450492e7b8973869f4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d8ae28c4ff54c7f849747b3df988e92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:41.808147 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787bd16518fc450492e7b8973869f4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.808147 6776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787bd16518fc450492e7b8973869f4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.808089 7058 tablet_service.cc:1467] Processing CreateTablet for tablet 787bd16518fc450492e7b8973869f4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d8ae28c4ff54c7f849747b3df988e92]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:41.808283 7058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787bd16518fc450492e7b8973869f4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:41.809775 7152 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: Bootstrap starting.
I20260924 15:27:41.809795 7148 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: Bootstrap starting.
I20260924 15:27:41.810230 7148 tablet_bootstrap.cc:654] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.810272 7152 tablet_bootstrap.cc:654] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.810645 7149 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: Bootstrap starting.
I20260924 15:27:41.810983 7148 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: No bootstrap required, opened a new log
I20260924 15:27:41.811017 7148 ts_tablet_manager.cc:1397] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:41.811064 7149 tablet_bootstrap.cc:654] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:41.811126 7148 raft_consensus.cc:348] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.811173 7148 raft_consensus.cc:374] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.811187 7148 raft_consensus.cc:729] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6918bf7d24c5ab5b0b32f15db63c6, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.811250 7148 consensus_queue.cc:260] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.811339 7148 ts_tablet_manager.cc:1428] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.811378 7154 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.811440 7154 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.811694 7152 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: No bootstrap required, opened a new log
I20260924 15:27:41.811753 7152 ts_tablet_manager.cc:1397] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:41.811895 7152 raft_consensus.cc:348] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.811976 7152 raft_consensus.cc:374] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.812003 7152 raft_consensus.cc:729] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b8e06a829a4793881ab01990d868c4, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.812077 7152 consensus_queue.cc:260] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.812187 7152 ts_tablet_manager.cc:1428] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:41.812219 7156 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.812306 7156 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.812505 7149 tablet_bootstrap.cc:492] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: No bootstrap required, opened a new log
I20260924 15:27:41.812547 7149 ts_tablet_manager.cc:1397] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:27:41.812816 7149 raft_consensus.cc:348] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.812884 7149 raft_consensus.cc:374] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:41.812901 7149 raft_consensus.cc:729] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf55583854d1428ba5bfe6d840d437f9, State: Initialized, Role: FOLLOWER
I20260924 15:27:41.812973 7149 consensus_queue.cc:260] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.813117 7149 ts_tablet_manager.cc:1428] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:41.813139 7155 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.813252 7155 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.865535 7154 raft_consensus.cc:482] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:41.865620 7154 raft_consensus.cc:504] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.865783 7154 leader_election.cc:288] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095), d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519)
I20260924 15:27:41.865959 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787bd16518fc450492e7b8973869f4a8" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9" is_pre_election: true
I20260924 15:27:41.866096 6937 raft_consensus.cc:2454] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 0.
I20260924 15:27:41.866093 7078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787bd16518fc450492e7b8973869f4a8" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b8e06a829a4793881ab01990d868c4" is_pre_election: true
I20260924 15:27:41.866187 7078 raft_consensus.cc:2454] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 0.
I20260924 15:27:41.866262 6729 leader_election.cc:302] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6918bf7d24c5ab5b0b32f15db63c6, bf55583854d1428ba5bfe6d840d437f9; no voters:
I20260924 15:27:41.866393 7154 raft_consensus.cc:2781] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:41.866436 7154 raft_consensus.cc:482] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:41.866485 7154 raft_consensus.cc:3037] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.867169 7154 raft_consensus.cc:504] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.867313 7154 leader_election.cc:288] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 election: Requested vote from peers bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095), d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519)
I20260924 15:27:41.867520 7078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787bd16518fc450492e7b8973869f4a8" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b8e06a829a4793881ab01990d868c4"
I20260924 15:27:41.867520 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787bd16518fc450492e7b8973869f4a8" candidate_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf55583854d1428ba5bfe6d840d437f9"
I20260924 15:27:41.867610 7078 raft_consensus.cc:3037] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.867610 6937 raft_consensus.cc:3037] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:41.868366 7078 raft_consensus.cc:2454] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 1.
I20260924 15:27:41.868366 6937 raft_consensus.cc:2454] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6918bf7d24c5ab5b0b32f15db63c6 in term 1.
I20260924 15:27:41.868602 6729 leader_election.cc:302] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6918bf7d24c5ab5b0b32f15db63c6, bf55583854d1428ba5bfe6d840d437f9; no voters:
I20260924 15:27:41.868703 7154 raft_consensus.cc:2781] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:41.868752 7154 raft_consensus.cc:686] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 LEADER]: Becoming Leader. State: Replica: 99f6918bf7d24c5ab5b0b32f15db63c6, State: Running, Role: LEADER
I20260924 15:27:41.868800 7154 consensus_queue.cc:237] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } }
I20260924 15:27:41.869156 7178 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: TxnStatusTablet state changed. Reason: New leader 99f6918bf7d24c5ab5b0b32f15db63c6. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.869271 7178 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:41.869458 7190 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:27:41.869503 6638 catalog_manager.cc:5495] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } health_report { overall_health: UNKNOWN } } }
I20260924 15:27:41.871021 7190 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:27:41.889832 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.887149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.5.98.193:37127)
0924 15:27:41.887200 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.887202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.887212 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:41.887302 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.887303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.887322 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.887386 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:41.887516 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.888177 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.888986 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.889201 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.889288 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.889499 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.889605 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.889686 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.889738 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.889761 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:27:41.889858 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.887088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:37127 (local address 127.0.0.1:52214)
0924 15:27:41.887165 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.887180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.887231 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.887391 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.887392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.887393 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:41.887452 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.887454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.888276 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.888277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.888913 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.888916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.889289 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.889290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.889358 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.889383 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.889672 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.889682 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.889734 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.889787 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:27:41.891119 6937 raft_consensus.cc:1261] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Refusing update from remote peer 99f6918bf7d24c5ab5b0b32f15db63c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.891172 7078 raft_consensus.cc:1261] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Refusing update from remote peer 99f6918bf7d24c5ab5b0b32f15db63c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.891395 7178 consensus_queue.cc:1035] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.891528 7154 consensus_queue.cc:1035] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.892025 7155 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: TxnStatusTablet state changed. Reason: New leader 99f6918bf7d24c5ab5b0b32f15db63c6. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.892124 7155 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.892246 7156 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: TxnStatusTablet state changed. Reason: New leader 99f6918bf7d24c5ab5b0b32f15db63c6. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.892316 7156 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.893440 7178 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.893520 7178 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:41.893785 7156 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.893869 7156 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P d5b8e06a829a4793881ab01990d868c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.894918 7178 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.894996 7178 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P 99f6918bf7d24c5ab5b0b32f15db63c6: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:41.895407 7155 tablet_replica.cc:439] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6918bf7d24c5ab5b0b32f15db63c6" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 37127 } } peers { permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 } } peers { permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 } } }
I20260924 15:27:41.895514 7155 tablet_replica.cc:442] T 787bd16518fc450492e7b8973869f4a8 P bf55583854d1428ba5bfe6d840d437f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:41.898257 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.895747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.5.98.193:37127)
0924 15:27:41.895932 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.895934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.895943 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:41.895963 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.895964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.895974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.896029 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:41.896160 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.896610 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.897656 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.897828 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.897877 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.898037 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.898062 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.898100 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.898176 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.898192 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260924 15:27:41.903591 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.900649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:50357 (local address 127.5.98.193:37127)
0924 15:27:41.900688 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.900690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.900885 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:27:41.900952 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.900953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.900960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.901003 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:41.901139 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.901703 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.902942 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.903119 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.903164 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.903372 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.903403 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.903446 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.903517 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.903531 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260924 15:27:41.903885 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.900602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:37127 (local address 127.5.98.193:50357)
0924 15:27:41.900810 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:41.900821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:41.900899 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:41.901008 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:41.901008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:41.901010 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:41.901063 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.901064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.901771 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.901772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.902847 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:41.902850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.903184 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.903185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:41.903256 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.903271 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:41.903465 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:41.903466 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:27:41.903515 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:27:41.903555 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260924 15:27:41.906190 7078 raft_consensus.cc:1261] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Refusing update from remote peer 99f6918bf7d24c5ab5b0b32f15db63c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.906291 6937 raft_consensus.cc:1261] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Refusing update from remote peer 99f6918bf7d24c5ab5b0b32f15db63c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:41.906487 7154 consensus_queue.cc:1035] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b8e06a829a4793881ab01990d868c4" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 37519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.909296 7178 consensus_queue.cc:1035] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf55583854d1428ba5bfe6d840d437f9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:41.919266 6638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:41.919361 6638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:41.920042 6638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07d020f2c90a4d238e82891e9c0ab9e3: Sending DeleteTablet for 3 replicas of tablet 746b619c348e4355ae1a1cd107999e2c
I20260924 15:27:41.920303 6775 tablet_service.cc:1514] Processing DeleteTablet for tablet 746b619c348e4355ae1a1cd107999e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:52200
I20260924 15:27:41.920395 7171 tablet_replica.cc:330] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: stopping tablet replica
I20260924 15:27:41.920444 7171 raft_consensus.cc:2229] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:41.920413 7058 tablet_service.cc:1514] Processing DeleteTablet for tablet 746b619c348e4355ae1a1cd107999e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:60406
I20260924 15:27:41.920552 7171 raft_consensus.cc:2258] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.920535 6917 tablet_service.cc:1514] Processing DeleteTablet for tablet 746b619c348e4355ae1a1cd107999e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:41 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:35034
I20260924 15:27:41.920621 7172 tablet_replica.cc:330] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: stopping tablet replica
I20260924 15:27:41.920626 7170 tablet_replica.cc:330] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: stopping tablet replica
I20260924 15:27:41.920675 7172 raft_consensus.cc:2229] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:41.920681 7170 raft_consensus.cc:2229] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:41.920739 7172 raft_consensus.cc:2258] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.920753 7170 raft_consensus.cc:2258] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:41.920851 7171 ts_tablet_manager.cc:1905] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:41.921147 7170 ts_tablet_manager.cc:1905] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:41.922783 7171 ts_tablet_manager.cc:1918] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:41.922813 7170 ts_tablet_manager.cc:1918] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:41.922868 7170 log.cc:1196] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/746b619c348e4355ae1a1cd107999e2c
I20260924 15:27:41.922869 7171 log.cc:1196] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/746b619c348e4355ae1a1cd107999e2c
I20260924 15:27:41.923095 7171 ts_tablet_manager.cc:1939] T 746b619c348e4355ae1a1cd107999e2c P 99f6918bf7d24c5ab5b0b32f15db63c6: Deleting consensus metadata
I20260924 15:27:41.923116 7170 ts_tablet_manager.cc:1939] T 746b619c348e4355ae1a1cd107999e2c P d5b8e06a829a4793881ab01990d868c4: Deleting consensus metadata
I20260924 15:27:41.923418 6621 catalog_manager.cc:4841] TS 99f6918bf7d24c5ab5b0b32f15db63c6 (127.5.98.193:37127): tablet 746b619c348e4355ae1a1cd107999e2c (table test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]) successfully deleted
I20260924 15:27:41.923437 6622 catalog_manager.cc:4841] TS d5b8e06a829a4793881ab01990d868c4 (127.5.98.195:37519): tablet 746b619c348e4355ae1a1cd107999e2c (table test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]) successfully deleted
I20260924 15:27:41.927886 7172 ts_tablet_manager.cc:1905] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 15:27:41 dist-test-slave-4g6f krb5kdc[6592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:41.929294 7172 ts_tablet_manager.cc:1918] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:41.929396 7172 log.cc:1196] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/746b619c348e4355ae1a1cd107999e2c
I20260924 15:27:41.929631 7172 ts_tablet_manager.cc:1939] T 746b619c348e4355ae1a1cd107999e2c P bf55583854d1428ba5bfe6d840d437f9: Deleting consensus metadata
I20260924 15:27:41.929950 6625 catalog_manager.cc:4841] TS bf55583854d1428ba5bfe6d840d437f9 (127.5.98.194:45095): tablet 746b619c348e4355ae1a1cd107999e2c (table test-table [id=f1d7ad7b24074a81a09bea2b75e8496b]) successfully deleted
I20260924 15:27:41.931826 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.921635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.5.98.193:37127)
0924 15:27:41.921669 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.921672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.921937 (+ 265us) server_negotiation.cc:407] Connection header received
0924 15:27:41.922033 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.922034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.922043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.922072 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.923584 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.924119 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.925190 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.925369 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.925434 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.928822 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.928830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.928830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.928838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.930641 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.931033 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.931035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.931036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.931086 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.931278 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.931278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.931279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.931397 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.931508 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.931607 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.931713 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:27:41.932052 6836 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52228
I20260924 15:27:41.939159 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:41.932917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.5.98.254:43347)
0924 15:27:41.932965 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:41.932966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:41.933154 (+ 188us) server_negotiation.cc:407] Connection header received
0924 15:27:41.933196 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:41.933197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:41.933208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:41.933269 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:41.933744 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.934511 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.935154 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:41.935475 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:41.935546 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:41.936601 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:41.936602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:41.936603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:41.936610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:41.938047 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.938357 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.938358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.938358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.938416 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:41.938644 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:41.938645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:41.938646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:41.938775 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:41.938872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:41.938950 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:27:41.939059 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 15:27:41.939386 6638 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55288
I20260924 15:27:41.940014 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6710
I20260924 15:27:41.947028 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6851
I20260924 15:27:41.951946 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6992
I20260924 15:27:41.957372 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 6605
2026-09-24T15:27:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2357 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:42 dist-test-slave-4g6f krb5kdc[7210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:42 dist-test-slave-4g6f krb5kdc[7210](info): set up 2 sockets
Sep 24 15:27:42 dist-test-slave-4g6f krb5kdc[7210](info): commencing operation
krb5kdc: starting...
W20260924 15:27:43.016383 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:43.135330 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33889
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:43.216957 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:43.217204 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:43.217240 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:43.218690 7223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:43.218782 7223 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:27:43.218809 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:43.218833 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:43.218863 7223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:43.218884 7223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:43.220515 7223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:33889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:43.220806 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:43.221046 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:43.226181 7229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.226155 7231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.226434 7223 server_base.cc:1034] running on GCE node
W20260924 15:27:43.226171 7228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.226749 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:43.227069 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:43.228214 7223 hybrid_clock.cc:648] HybridClock initialized: now 1790263663228192 us; error 43 us; skew 500 ppm
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:43.230695 7223 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:43.231179 7223 webserver.cc:460] Webserver started at http://127.5.98.254:35547/ using document root <none> and password file <none>
I20260924 15:27:43.231427 7223 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:43.231504 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:43.231698 7223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:43.232781 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "fad5621f60794d8ea3d4a03a8ee6597b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "734f36d231e8a22df66232deca2babc6"
server_key_iv: "828eaac42768307c4727783a76b3d556"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.233146 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "fad5621f60794d8ea3d4a03a8ee6597b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "734f36d231e8a22df66232deca2babc6"
server_key_iv: "828eaac42768307c4727783a76b3d556"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.234452 7223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:43.235174 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.235347 7223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:43.235451 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "fad5621f60794d8ea3d4a03a8ee6597b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "734f36d231e8a22df66232deca2babc6"
server_key_iv: "828eaac42768307c4727783a76b3d556"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.235531 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:43.273667 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:43.275636 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:43.275837 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:43.280483 7223 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:33889
I20260924 15:27:43.280542 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:33889 every 8 connection(s)
I20260924 15:27:43.280921 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:43.281597 7302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.283947 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Bootstrap starting.
I20260924 15:27:43.284509 7302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.284746 7302 log.cc:824] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:43.285396 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: No bootstrap required, opened a new log
I20260924 15:27:43.287649 7302 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } }
I20260924 15:27:43.287863 7302 raft_consensus.cc:374] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.287886 7302 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad5621f60794d8ea3d4a03a8ee6597b, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.288021 7302 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } }
I20260924 15:27:43.288094 7302 raft_consensus.cc:388] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:43.288116 7302 raft_consensus.cc:482] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:43.288146 7302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:43.288628 7302 raft_consensus.cc:504] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } }
I20260924 15:27:43.288712 7302 leader_election.cc:302] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fad5621f60794d8ea3d4a03a8ee6597b; no voters:
I20260924 15:27:43.288851 7302 leader_election.cc:288] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:43.288880 7305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:43.289007 7305 raft_consensus.cc:686] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [term 1 LEADER]: Becoming Leader. State: Replica: fad5621f60794d8ea3d4a03a8ee6597b, State: Running, Role: LEADER
I20260924 15:27:43.289119 7302 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:43.289165 7305 consensus_queue.cc:237] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } }
I20260924 15:27:43.289537 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fad5621f60794d8ea3d4a03a8ee6597b. Latest consensus state: current_term: 1 leader_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } } }
I20260924 15:27:43.289569 7306 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad5621f60794d8ea3d4a03a8ee6597b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33889 } } }
I20260924 15:27:43.289718 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:43.289819 7306 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:43.290102 7314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:43.290580 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 7223
I20260924 15:27:43.290668 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:43.290688 7314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:43.290836 5515 external_mini_cluster.cc:1428] Setting key 59651cf81bc28807dc4818f4e00181ec
I20260924 15:27:43.292243 7314 catalog_manager.cc:1347] Generated new cluster ID: 69ee45e4e96f47c0bcc4f4e2945a5553
I20260924 15:27:43.292290 7314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:43.297796 7314 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:43.298506 7314 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:43.302618 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.291448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.5.98.254:33889)
0924 15:27:43.291614 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.291623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.291919 (+ 296us) server_negotiation.cc:407] Connection header received
0924 15:27:43.292046 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.292054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.292109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.292236 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.293101 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.294468 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.295596 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.296204 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.296311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.299085 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.299089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.299089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.299100 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.300948 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.301287 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.301290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.301290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.301375 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.301549 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.301550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.301550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.301780 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.301903 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.302269 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.302398 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260924 15:27:43.306296 7314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Generated new TSK 0
I20260924 15:27:43.306478 7314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:43.338122 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33889
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:43.421103 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:43.421365 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:43.421415 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:43.422551 7328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:43.422616 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:43.422652 7328 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:27:43.422679 7328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:43.422713 7328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:43.424163 7328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:43.424445 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:43.424674 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:43.429636 7336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.429756 7333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.429786 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.429811 7328 server_base.cc:1034] running on GCE node
I20260924 15:27:43.430078 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:43.430310 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:43.431474 7328 hybrid_clock.cc:648] HybridClock initialized: now 1790263663431472 us; error 55 us; skew 500 ppm
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:43.433951 7328 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:43.434420 7328 webserver.cc:460] Webserver started at http://127.5.98.193:42087/ using document root <none> and password file <none>
I20260924 15:27:43.434670 7328 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:43.434739 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:43.434900 7328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:43.436048 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "3963434a5cd84c1aad3fe16961bf25f4"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "7169c9c15991792f1329b935fb0e80b6"
server_key_iv: "c465d13eb192b7ab02713a180984cc1e"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.436419 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "3963434a5cd84c1aad3fe16961bf25f4"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "7169c9c15991792f1329b935fb0e80b6"
server_key_iv: "c465d13eb192b7ab02713a180984cc1e"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.437690 7328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.438529 7343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.438735 7328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.438839 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "3963434a5cd84c1aad3fe16961bf25f4"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "7169c9c15991792f1329b935fb0e80b6"
server_key_iv: "c465d13eb192b7ab02713a180984cc1e"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.438915 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:43.449767 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:43.451735 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:43.451901 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:43.452520 7328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:43.452569 7328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.452600 7328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:43.452620 7328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:43.459878 7328 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:36133
I20260924 15:27:43.459969 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:36133 every 8 connection(s)
I20260924 15:27:43.460287 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:43.462636 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.452931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:48509 (local address 127.5.98.254:33889)
0924 15:27:43.453008 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.453010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.453276 (+ 266us) server_negotiation.cc:407] Connection header received
0924 15:27:43.453829 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.453831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.453844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.453896 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.454596 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.455568 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.456731 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.456965 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.457060 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.459582 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.459584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.459584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.459594 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.461368 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.461744 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.461745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.461745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.461799 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.462068 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.462069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.462069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.462177 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.462245 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.462502 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.462570 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:27:43.462772 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.452913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.193:48509)
0924 15:27:43.453123 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.453164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.453715 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.454009 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.454014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.454287 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.454488 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.454493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.455709 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.455711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.456619 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.456622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.457085 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.457086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.457214 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.457798 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.457812 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.459444 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.461456 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.461461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.461463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.461673 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.461878 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.461879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.461880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.461998 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.462249 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.462261 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.462378 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.462468 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.462585 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":134,"threads_started":1}
I20260924 15:27:43.463246 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 7328
I20260924 15:27:43.463339 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:43.463336 7460 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33889
I20260924 15:27:43.463420 7460 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:43.463554 5515 external_mini_cluster.cc:1428] Setting key 5b43e3eb73bb53053903931fd124aa9c
I20260924 15:27:43.463564 7460 heartbeater.cc:511] Master 127.5.98.254:33889 requested a full tablet report, sending...
I20260924 15:27:43.464241 7255 ts_manager.cc:194] Registered new tserver with Master: 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133)
I20260924 15:27:43.464921 7255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:48509
I20260924 15:27:43.472580 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.465315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.193:35467)
0924 15:27:43.466159 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.466171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.466232 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.466368 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.466368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.466531 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.466588 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.466589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.467618 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.467619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.468215 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.468217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.468744 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.468746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.468806 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.469295 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.469299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.469593 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.471489 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.471490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.471490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.471749 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.471985 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.471985 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.471986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.472052 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.472332 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.472333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.472368 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.472418 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.472492 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":753,"threads_started":1}
I20260924 15:27:43.472651 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.465354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:35467 (local address 127.5.98.254:33889)
0924 15:27:43.465398 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.465400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.466248 (+ 848us) server_negotiation.cc:407] Connection header received
0924 15:27:43.466279 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.466280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.466293 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.466366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.466665 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.467516 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.468301 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.468662 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.468747 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.469687 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.469689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.469690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.469699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.471374 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.471829 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.471830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.471831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.471910 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.472121 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.472122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.472123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.472260 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.472347 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.472447 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.472530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:43.491313 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33889
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:43.577174 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:43.577359 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:43.577383 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:43.578461 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:43.578500 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:43.578519 7469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:27:43.578531 7469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:43.578547 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:43.579931 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:43.580113 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:43.580278 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:43.585312 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.585338 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.585464 7469 server_base.cc:1034] running on GCE node
W20260924 15:27:43.585583 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.585810 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:43.586050 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:43.587189 7469 hybrid_clock.cc:648] HybridClock initialized: now 1790263663587172 us; error 33 us; skew 500 ppm
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:43.589715 7469 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:43.590175 7469 webserver.cc:460] Webserver started at http://127.5.98.194:39963/ using document root <none> and password file <none>
I20260924 15:27:43.590368 7469 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:43.590405 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:43.590488 7469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:43.591509 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "fa396baea266e8652c9c299ab365df5c"
server_key_iv: "039915b43b49776e061316946eb118d1"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.591866 7469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "fa396baea266e8652c9c299ab365df5c"
server_key_iv: "039915b43b49776e061316946eb118d1"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.593101 7469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.593793 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.593967 7469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:43.594055 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "fa396baea266e8652c9c299ab365df5c"
server_key_iv: "039915b43b49776e061316946eb118d1"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.594131 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:43.606148 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:43.608156 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:43.608309 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:43.608918 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:43.608969 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.609004 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:43.609023 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:43.616006 7469 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:42715
I20260924 15:27:43.616082 7599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:42715 every 8 connection(s)
I20260924 15:27:43.616451 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:43.618685 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.609373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:35221 (local address 127.5.98.254:33889)
0924 15:27:43.609444 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.609448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.609646 (+ 198us) server_negotiation.cc:407] Connection header received
0924 15:27:43.610130 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.610132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.610147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.610233 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.611012 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.611856 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.612860 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.613077 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.613156 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.616094 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.616095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.616096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.616104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.617562 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.617929 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.617930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.617930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.617978 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.618193 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.618193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.618194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.618288 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.618337 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.618563 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.618623 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:43.618832 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.609352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.194:35221)
0924 15:27:43.609510 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.609552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.610038 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.610321 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.610326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.610604 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.610786 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.610790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.611959 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.611960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.612762 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.612764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.613175 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.613176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.613295 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.613822 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.613831 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.615977 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.617638 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.617643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.617644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.617854 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.618044 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.618044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.618045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.618139 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.618343 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.618347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.618455 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.618538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.618650 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 15:27:43.619376 7600 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33889
I20260924 15:27:43.619467 7600 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:43.619644 7600 heartbeater.cc:511] Master 127.5.98.254:33889 requested a full tablet report, sending...
I20260924 15:27:43.620060 7254 ts_manager.cc:194] Registered new tserver with Master: f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:43.620558 7254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:35221
I20260924 15:27:43.625874 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.620447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:52035 (local address 127.5.98.254:33889)
0924 15:27:43.620496 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.620499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.620664 (+ 165us) server_negotiation.cc:407] Connection header received
0924 15:27:43.620687 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.620687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.620697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.620760 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.621092 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.621550 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.622255 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.622474 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.622550 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.623437 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.623438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.623438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.623445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.624732 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.625080 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.625081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.625081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.625136 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.625336 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.625337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.625338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.625479 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.625557 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.625710 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.625783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:27:43.625880 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.620379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.194:52035)
0924 15:27:43.620551 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.620568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.620616 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.620778 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.620779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.620949 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.621005 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.621007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.621640 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.621641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.622160 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.622162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.622550 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.622551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.622628 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.623059 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.623064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.623334 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.624825 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.624826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.624827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.624990 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.625204 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.625205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.625206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.625264 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.625550 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.625551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.625597 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.625686 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.625785 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260924 15:27:43.626073 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 7469
I20260924 15:27:43.626171 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:43.626372 5515 external_mini_cluster.cc:1428] Setting key d0134184884cc24f06b603b0994ff576
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:43.650651 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33889
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:43.731940 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:43.732239 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:43.732281 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:43.734454 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:43.734517 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:43.734553 7609 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:27:43.734575 7609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:27:43.734617 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:43.736593 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:33889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:43.736856 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:43.737099 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:43.743616 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.743609 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:43.743592 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:43.743830 7609 server_base.cc:1034] running on GCE node
I20260924 15:27:43.744105 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:43.744374 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:43.745527 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790263663745500 us; error 44 us; skew 500 ppm
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:43.747943 7609 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:43.748369 7609 webserver.cc:460] Webserver started at http://127.5.98.195:42905/ using document root <none> and password file <none>
I20260924 15:27:43.748605 7609 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:43.748674 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:43.748828 7609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:43.749878 7609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "6022c81c34f84a51931f685c4d19a153"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "95e4a803fe5dbf14ebfd9dd8e56c4d65"
server_key_iv: "8eef2299414a8fec1ba9e953bb2bf961"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.750242 7609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "6022c81c34f84a51931f685c4d19a153"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "95e4a803fe5dbf14ebfd9dd8e56c4d65"
server_key_iv: "8eef2299414a8fec1ba9e953bb2bf961"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.751626 7609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:43.752596 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.752780 7609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.752882 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "6022c81c34f84a51931f685c4d19a153"
format_stamp: "Formatted at 2026-09-24 15:27:43 on dist-test-slave-4g6f"
server_key: "95e4a803fe5dbf14ebfd9dd8e56c4d65"
server_key_iv: "8eef2299414a8fec1ba9e953bb2bf961"
server_key_version: "encryptionkey@0"
I20260924 15:27:43.752974 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:43.762390 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:43.764564 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:43.764729 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:43.765340 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:43.765413 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.765450 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:43.765475 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:43.772663 7609 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42635
I20260924 15:27:43.772742 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42635 every 8 connection(s)
I20260924 15:27:43.773094 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:43.775468 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 7609
I20260924 15:27:43.775579 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.765845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:58219 (local address 127.5.98.254:33889)
0924 15:27:43.765932 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.765935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.766124 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:27:43.766603 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.766605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.766619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.766706 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.767436 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.768309 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.769422 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.769769 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.769851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.772573 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.772575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.772575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.772583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.774165 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.774579 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.774581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.774581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.774643 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.774898 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.774899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.774899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.775023 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.775099 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.775399 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.775483 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:27:43.775588 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:43.775797 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.765824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.195:58219)
0924 15:27:43.765990 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.766031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.766508 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.766810 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.766814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.767132 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.767325 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.767331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.768446 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.768448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.769312 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.769315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.769883 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.769884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.770020 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.770511 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.770519 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.772486 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.774271 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.774277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.774280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.774500 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.774712 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.774713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.774713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.774822 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.775091 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.775098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.775244 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.775374 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.775547 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260924 15:27:43.775849 5515 external_mini_cluster.cc:1428] Setting key bfce8229d477953ec1d7b7f2cf46674f
I20260924 15:27:43.777125 7740 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33889
I20260924 15:27:43.777182 7740 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:43.777384 7740 heartbeater.cc:511] Master 127.5.98.254:33889 requested a full tablet report, sending...
I20260924 15:27:43.777733 7254 ts_manager.cc:194] Registered new tserver with Master: 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635)
I20260924 15:27:43.778242 7254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:58219
I20260924 15:27:43.782979 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.777283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:33653 (local address 127.5.98.254:33889)
0924 15:27:43.777342 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.777344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.777501 (+ 157us) server_negotiation.cc:407] Connection header received
0924 15:27:43.777562 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.777563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.777573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.777636 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.778028 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.778710 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.779449 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.779622 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.779751 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.780573 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.780574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.780574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.780580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.782052 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.782385 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.782386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.782386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.782433 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.782624 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.782624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.782624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.782715 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.782758 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.782868 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.782919 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:27:43.782995 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.777284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.5.98.195:33653)
0924 15:27:43.777444 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.777454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.777513 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.777643 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.777643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.777877 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.777952 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.777954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.778801 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.778802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.779364 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.779366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.779769 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.779769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.779824 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.780178 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.780183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.780458 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.782130 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.782131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.782132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.782286 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.782521 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.782521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.782522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.782575 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.782764 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.782765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.782800 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.782846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.782918 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":15,"thread_start_us":110,"threads_started":1}
I20260924 15:27:43.787554 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:43.810681 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.803364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.5.98.254:33889)
0924 15:27:43.803435 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.803437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.803447 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:43.803510 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.803511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.803521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.803573 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.804174 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.804689 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.805396 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.805603 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.805661 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.808307 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.808308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.808308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.808315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.809744 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.810061 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.810062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.810062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.810112 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.810277 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.810278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.810278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.810375 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.810448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.810532 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.810589 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:27:43.812474 7254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:43.814047 7254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:43.820950 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.5.98.195:42635)
0924 15:27:43.817117 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.817121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.817347 (+ 226us) server_negotiation.cc:407] Connection header received
0924 15:27:43.817391 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.817394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.817409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.817509 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:43.817620 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818307 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.819888 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820483 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820583 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820648 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.820796 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.820842 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260924 15:27:43.820963 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36133 (local address 127.0.0.1:53954)
0924 15:27:43.816936 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.816946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.816999 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.817283 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.817284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.817286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:43.817320 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.817320 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818618 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.818619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.819763 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.819766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820685 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.820752 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820756 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.820787 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.820855 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":21,"thread_start_us":74,"threads_started":1}
I20260924 15:27:43.821087 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.5.98.193:36133)
0924 15:27:43.817117 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.817122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.817131 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:43.817167 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.817168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.817183 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.817392 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:43.817452 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818537 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.819851 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820590 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820834 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820897 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.820923 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.820965 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":74,"threads_started":1}
I20260924 15:27:43.821203 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42715 (local address 127.0.0.1:54302)
0924 15:27:43.816729 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.816773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.816838 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.817163 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.817166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.817168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:43.817240 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.817242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818644 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.818645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.819641 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.819643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820724 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.820786 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820790 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.820834 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.820875 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260924 15:27:43.821519 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.0.0.1:59612)
0924 15:27:43.817225 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.817236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.817286 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.817506 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.817507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.817509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:43.817563 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.817565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818421 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.818422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.819802 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.819804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820585 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.820664 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820679 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.820778 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.820928 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":82,"threads_started":1}
I20260924 15:27:43.821744 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 (DEFAULT_TABLE table=test-table [id=6a881b99d9e245d596fe29ea3bbbf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:43.822135 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6411fcadf045d680dbecbb0e303a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.822232 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 (DEFAULT_TABLE table=test-table [id=6a881b99d9e245d596fe29ea3bbbf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:43.822562 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6411fcadf045d680dbecbb0e303a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.823865 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.816755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.5.98.194:42715)
0924 15:27:43.816902 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.816908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.816921 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:27:43.816972 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.816974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.817001 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.817080 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:43.817325 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.818520 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.819781 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.820554 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.820631 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.820696 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.823742 (+ 3046us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.823794 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260924 15:27:43.824427 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 (DEFAULT_TABLE table=test-table [id=6a881b99d9e245d596fe29ea3bbbf613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:43.824730 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6411fcadf045d680dbecbb0e303a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.825970 7763 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Bootstrap starting.
I20260924 15:27:43.826200 7764 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Bootstrap starting.
I20260924 15:27:43.826779 7764 tablet_bootstrap.cc:654] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.827023 7763 tablet_bootstrap.cc:654] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.827131 7764 log.cc:824] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:43.827390 7763 log.cc:824] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:43.828018 7764 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: No bootstrap required, opened a new log
I20260924 15:27:43.828096 7764 ts_tablet_manager.cc:1397] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:43.828145 7767 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Bootstrap starting.
I20260924 15:27:43.828307 7763 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: No bootstrap required, opened a new log
I20260924 15:27:43.828374 7763 ts_tablet_manager.cc:1397] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:43.828822 7767 tablet_bootstrap.cc:654] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.829152 7767 log.cc:824] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:43.829874 7767 tablet_bootstrap.cc:492] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: No bootstrap required, opened a new log
I20260924 15:27:43.829959 7767 ts_tablet_manager.cc:1397] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:43.829933 7764 raft_consensus.cc:348] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.830102 7764 raft_consensus.cc:374] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.830144 7764 raft_consensus.cc:729] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963434a5cd84c1aad3fe16961bf25f4, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.830277 7764 consensus_queue.cc:260] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.830556 7764 ts_tablet_manager.cc:1428] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:27:43.830590 7460 heartbeater.cc:503] Master 127.5.98.254:33889 was elected leader, sending a full tablet report...
I20260924 15:27:43.830546 7763 raft_consensus.cc:348] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.830709 7763 raft_consensus.cc:374] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.830749 7763 raft_consensus.cc:729] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.830858 7763 consensus_queue.cc:260] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.831122 7763 ts_tablet_manager.cc:1428] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:43.831148 7740 heartbeater.cc:503] Master 127.5.98.254:33889 was elected leader, sending a full tablet report...
I20260924 15:27:43.832409 7767 raft_consensus.cc:348] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.832549 7767 raft_consensus.cc:374] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.832593 7767 raft_consensus.cc:729] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.832701 7767 consensus_queue.cc:260] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.832962 7767 ts_tablet_manager.cc:1428] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:43.833022 7600 heartbeater.cc:503] Master 127.5.98.254:33889 was elected leader, sending a full tablet report...
I20260924 15:27:43.834242 7770 raft_consensus.cc:482] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:43.834311 7770 raft_consensus.cc:504] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.834566 7770 leader_election.cc:288] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715), 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133)
I20260924 15:27:43.837438 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.834777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:58939 (local address 127.5.98.194:42715)
0924 15:27:43.834828 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.834830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.834974 (+ 144us) server_negotiation.cc:407] Connection header received
0924 15:27:43.835034 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.835035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.835059 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.835130 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:43.835302 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.835836 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.836735 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.837090 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.837145 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.837173 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.837324 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.837367 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:27:43.837481 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.834787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:60401 (local address 127.5.98.193:36133)
0924 15:27:43.834968 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.834969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.834977 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:43.834999 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.835000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.835009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.835060 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:43.835230 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.835814 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.836737 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.837097 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.837183 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.837191 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.837376 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.837418 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 15:27:43.837457 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.834738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42715 (local address 127.5.98.195:58939)
0924 15:27:43.834886 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.834895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.834966 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.835139 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.835140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.835142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:43.835230 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.835231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.835927 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.835928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.836641 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.836643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.837171 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.837172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.837245 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.837249 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.837303 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.837331 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":22,"thread_start_us":84,"threads_started":1}
I20260924 15:27:43.837728 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.834738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36133 (local address 127.5.98.195:60401)
0924 15:27:43.834827 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.834839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.834891 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.835078 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.835079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.835081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:43.835161 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.835163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.835890 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.835891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.836636 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.836638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.837197 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.837198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.837280 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.837283 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.837358 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.837398 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":2}
I20260924 15:27:43.838023 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6411fcadf045d680dbecbb0e303a92" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" is_pre_election: true
I20260924 15:27:43.838104 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6411fcadf045d680dbecbb0e303a92" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4" is_pre_election: true
I20260924 15:27:43.838150 7554 raft_consensus.cc:2454] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:43.838215 7414 raft_consensus.cc:2454] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:43.838356 7626 leader_election.cc:302] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:43.838517 7770 raft_consensus.cc:2781] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:43.838570 7770 raft_consensus.cc:482] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:43.838593 7770 raft_consensus.cc:3037] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:43.839390 7770 raft_consensus.cc:504] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.839591 7770 leader_election.cc:288] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Requested vote from peers f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715), 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133)
I20260924 15:27:43.839692 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6411fcadf045d680dbecbb0e303a92" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4"
I20260924 15:27:43.839690 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6411fcadf045d680dbecbb0e303a92" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
I20260924 15:27:43.839769 7414 raft_consensus.cc:3037] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:43.839769 7554 raft_consensus.cc:3037] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:43.840582 7414 raft_consensus.cc:2454] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:43.840582 7554 raft_consensus.cc:2454] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:43.840736 7626 leader_election.cc:302] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:43.840826 7770 raft_consensus.cc:2781] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:43.840993 7770 raft_consensus.cc:686] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Becoming Leader. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Running, Role: LEADER
I20260924 15:27:43.841094 7770 consensus_queue.cc:237] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } }
I20260924 15:27:43.841775 7255 catalog_manager.cc:5495] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6022c81c34f84a51931f685c4d19a153 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:27:43 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
W20260924 15:27:43.867111 7601 tablet.cc:2369] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:43.867766 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.860419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.5.98.254:33889)
0924 15:27:43.860505 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:43.860508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:43.860618 (+ 110us) server_negotiation.cc:407] Connection header received
0924 15:27:43.860703 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:43.860704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:43.860714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:43.860783 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:43.861248 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.861769 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.862483 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.862728 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.862810 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.865010 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:43.865011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:43.865011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:43.865018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:43.866256 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.866603 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.866603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.866604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.866652 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:43.867094 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:43.867095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:43.867095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:43.867252 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:43.867343 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:43.867542 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:27:43.867646 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:27:43.867867 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:43.860336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33889 (local address 127.0.0.1:44086)
0924 15:27:43.860536 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:43.860547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:43.860639 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:43.860789 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:43.860790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:43.861118 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:43.861162 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.861164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.861853 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.861854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.862400 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:43.862402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:43.862810 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:43.862812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:43.862865 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:43.863423 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:43.863428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:43.864907 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:43.866331 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.866338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.866340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.866538 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.866713 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:43.866714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:43.866714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:43.867016 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:43.867335 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:43.867343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:43.867393 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 15:27:43.867514 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 15:27:43.867734 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":87,"thread_start_us":119,"threads_started":1}
I20260924 15:27:43.869949 7255 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:44086:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:27:43.870163 7255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:43.871711 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 4d57936604e04c18842a34b78c2e8608 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ceda65477904354bf260b59d48df0ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:43.871711 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 4d57936604e04c18842a34b78c2e8608 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ceda65477904354bf260b59d48df0ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:43.871742 7674 tablet_service.cc:1467] Processing CreateTablet for tablet 4d57936604e04c18842a34b78c2e8608 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ceda65477904354bf260b59d48df0ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:27:43.871883 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d57936604e04c18842a34b78c2e8608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.871883 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d57936604e04c18842a34b78c2e8608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.871933 7674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d57936604e04c18842a34b78c2e8608. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:43.873382 7764 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: Bootstrap starting.
I20260924 15:27:43.873616 7763 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: Bootstrap starting.
I20260924 15:27:43.874004 7763 tablet_bootstrap.cc:654] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.874038 7764 tablet_bootstrap.cc:654] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.874356 7767 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: Bootstrap starting.
I20260924 15:27:43.874632 7763 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: No bootstrap required, opened a new log
I20260924 15:27:43.874657 7764 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: No bootstrap required, opened a new log
I20260924 15:27:43.874699 7763 ts_tablet_manager.cc:1397] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.874711 7764 ts_tablet_manager.cc:1397] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:43.874867 7763 raft_consensus.cc:348] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.874881 7764 raft_consensus.cc:348] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.874953 7763 raft_consensus.cc:374] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.874965 7764 raft_consensus.cc:374] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.874985 7763 raft_consensus.cc:729] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.874993 7764 raft_consensus.cc:729] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963434a5cd84c1aad3fe16961bf25f4, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.875029 7763 consensus_queue.cc:260] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.875059 7767 tablet_bootstrap.cc:654] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:43.875072 7764 consensus_queue.cc:260] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.875159 7763 ts_tablet_manager.cc:1428] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:43.875201 7764 ts_tablet_manager.cc:1428] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.875275 7770 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:43.875277 7769 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:43.875342 7770 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:43.875366 7769 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:43.875777 7767 tablet_bootstrap.cc:492] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: No bootstrap required, opened a new log
I20260924 15:27:43.875833 7767 ts_tablet_manager.cc:1397] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:43.875983 7767 raft_consensus.cc:348] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.876068 7767 raft_consensus.cc:374] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:43.876092 7767 raft_consensus.cc:729] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Initialized, Role: FOLLOWER
I20260924 15:27:43.876149 7767 consensus_queue.cc:260] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:43.876235 7767 ts_tablet_manager.cc:1428] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:43.876318 7771 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:43.876417 7771 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 15:27:43.961052 7461 tablet.cc:2369] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:44.023852 7741 tablet.cc:2369] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:44.219616 7770 consensus_queue.cc:1035] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.222095 7770 consensus_queue.cc:1035] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.234035 7769 raft_consensus.cc:482] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:44.234117 7769 raft_consensus.cc:504] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.234366 7769 leader_election.cc:288] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715), 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635)
I20260924 15:27:44.237128 7770 raft_consensus.cc:482] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:44.237221 7770 raft_consensus.cc:504] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.237360 7770 leader_election.cc:288] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:44.237536 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4" is_pre_election: true
I20260924 15:27:44.237558 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" is_pre_election: true
I20260924 15:27:44.237617 7414 raft_consensus.cc:2454] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:44.237632 7554 raft_consensus.cc:2454] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:44.237756 7626 leader_election.cc:302] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:44.237846 7770 raft_consensus.cc:2781] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:44.237875 7770 raft_consensus.cc:482] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:44.237892 7770 raft_consensus.cc:3037] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.238372 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.234587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42715 (local address 127.5.98.193:38347)
0924 15:27:44.234719 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:44.234732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:44.234798 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:44.234960 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:44.234961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:44.234963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:44.235028 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.235031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.235990 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.235995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.237300 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.237304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.238105 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.238106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.238176 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.238180 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:44.238243 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:27:44.238284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":64,"thread_start_us":87,"threads_started":1}
I20260924 15:27:44.238421 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.234671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:38347 (local address 127.5.98.194:42715)
0924 15:27:44.234722 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:44.234724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:44.234862 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:27:44.234890 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:44.234891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:44.234901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:44.234949 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:44.235126 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.235904 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.237428 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.238001 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.238090 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.238102 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:44.238269 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 15:27:44.238318 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:27:44.238538 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.234847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:34613 (local address 127.5.98.195:42635)
0924 15:27:44.234910 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:44.234912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:44.234983 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:27:44.235036 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:44.235037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:44.235047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:44.235133 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:44.235281 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.236014 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.237462 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.238044 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.238120 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.238128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:44.238415 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 15:27:44.238464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:27:44.238849 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.234793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.5.98.193:34613)
0924 15:27:44.234912 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:44.234921 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:44.234975 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:44.235150 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:44.235150 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:44.235152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:44.235215 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.235216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.236080 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.236080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.237300 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.237304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.238150 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.238151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.238312 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.238317 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:27:44.238680 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 15:27:44.238734 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260924 15:27:44.238831 7770 raft_consensus.cc:504] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.238962 7770 leader_election.cc:288] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Requested vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:44.238977 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "3963434a5cd84c1aad3fe16961bf25f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" is_pre_election: true
I20260924 15:27:44.239069 7554 raft_consensus.cc:2454] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3963434a5cd84c1aad3fe16961bf25f4 in term 0.
I20260924 15:27:44.239116 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4"
I20260924 15:27:44.239152 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "3963434a5cd84c1aad3fe16961bf25f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6022c81c34f84a51931f685c4d19a153" is_pre_election: true
I20260924 15:27:44.239197 7414 raft_consensus.cc:3037] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.239271 7694 raft_consensus.cc:2379] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3963434a5cd84c1aad3fe16961bf25f4 in current term 1: Already voted for candidate 6022c81c34f84a51931f685c4d19a153 in this term.
I20260924 15:27:44.239488 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d57936604e04c18842a34b78c2e8608" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
I20260924 15:27:44.239554 7554 raft_consensus.cc:3037] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.239784 7345 leader_election.cc:302] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3963434a5cd84c1aad3fe16961bf25f4, f2cdf979af2d48dd9b93cc2b05819f4b; no voters: 6022c81c34f84a51931f685c4d19a153
I20260924 15:27:44.240017 7414 raft_consensus.cc:2454] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:44.240200 7625 leader_election.cc:302] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963434a5cd84c1aad3fe16961bf25f4, 6022c81c34f84a51931f685c4d19a153; no voters:
I20260924 15:27:44.240314 7770 raft_consensus.cc:2781] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:44.240345 7554 raft_consensus.cc:2454] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:44.240386 7770 raft_consensus.cc:686] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Becoming Leader. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Running, Role: LEADER
I20260924 15:27:44.240453 7769 raft_consensus.cc:2741] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 15:27:44.240444 7770 consensus_queue.cc:237] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.240826 7773 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: TxnStatusTablet state changed. Reason: New leader 6022c81c34f84a51931f685c4d19a153. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.240939 7773 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:44.241138 7792 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:27:44.241266 7792 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:27:44.241170 7254 catalog_manager.cc:5495] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6022c81c34f84a51931f685c4d19a153 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } health_report { overall_health: HEALTHY } } }
I20260924 15:27:44.317408 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.314516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.5.98.195:42635)
0924 15:27:44.314570 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:44.314572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:44.314581 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:44.314683 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:44.314683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:44.314700 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:44.314761 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:44.314963 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.315450 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.316577 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.316769 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.316834 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.317076 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:44.317169 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:44.317224 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:44.317327 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 15:27:44.317347 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:27:44.317466 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.314427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.0.0.1:59622)
0924 15:27:44.314523 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:44.314536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:44.314588 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:44.314804 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:44.314805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:44.314807 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:44.314889 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.314892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.315550 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.315552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.316458 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.316462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.316862 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.316863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.316944 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.316964 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:44.317242 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:44.317253 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 15:27:44.317326 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:27:44.317382 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 15:27:44.319092 7414 raft_consensus.cc:1261] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:44.319092 7554 raft_consensus.cc:1261] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:44.319360 7773 consensus_queue.cc:1035] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.319561 7770 consensus_queue.cc:1035] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.320449 7769 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: TxnStatusTablet state changed. Reason: New leader 6022c81c34f84a51931f685c4d19a153. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.320585 7769 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:44.320740 7771 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: TxnStatusTablet state changed. Reason: New leader 6022c81c34f84a51931f685c4d19a153. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.320847 7771 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:44.321879 7770 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.322021 7770 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:44.321887 7773 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.322069 7773 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 6022c81c34f84a51931f685c4d19a153: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:27:44.322343 7771 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.322389 7788 mvcc.cc:205] Tried to move back new op lower bound from 7332919969048698880 to 7332919968729337856. Current Snapshot: MvccSnapshot[applied={T|T < 7332919969048698880}]
I20260924 15:27:44.322439 7771 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P f2cdf979af2d48dd9b93cc2b05819f4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:44.323071 7769 tablet_replica.cc:439] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } } }
I20260924 15:27:44.323113 7786 mvcc.cc:205] Tried to move back new op lower bound from 7332919969048698880 to 7332919968729337856. Current Snapshot: MvccSnapshot[applied={T|T < 7332919969048698880}]
I20260924 15:27:44.323159 7769 tablet_replica.cc:442] T 4d57936604e04c18842a34b78c2e8608 P 3963434a5cd84c1aad3fe16961bf25f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:27:44.327086 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.324343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.5.98.195:42635)
0924 15:27:44.324386 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:44.324388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:44.324560 (+ 172us) server_negotiation.cc:407] Connection header received
0924 15:27:44.324610 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:44.324611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:44.324622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:44.324675 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:44.324808 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.325265 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.326415 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.326596 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.326660 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.326850 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:44.326877 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:44.326922 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:44.327018 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:27:44.327037 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:44.334040 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.329595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:56515 (local address 127.5.98.195:42635)
0924 15:27:44.329665 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:44.329667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:44.329733 (+ 66us) server_negotiation.cc:407] Connection header received
0924 15:27:44.329814 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:44.329815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:44.329822 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:44.329859 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:44.330006 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.330456 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.333241 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.333418 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.333514 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.333756 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:44.333787 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:44.333832 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:44.333948 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:27:44.333965 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:27:44.334167 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:44.329503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.5.98.195:56515)
0924 15:27:44.329670 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:44.329682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:44.329751 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:44.329871 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:44.329872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:44.329874 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:44.329938 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.329939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.330543 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.330544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.333128 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:44.333132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:44.333542 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:44.333544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:44.333617 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:44.333633 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:44.333860 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:44.333862 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:27:44.333934 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:27:44.333982 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260924 15:27:44.350154 7255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:44.350282 7255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:44.351034 7255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Sending DeleteTablet for 3 replicas of tablet 0a6411fcadf045d680dbecbb0e303a92
I20260924 15:27:44.351392 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:44 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260924 15:27:44.351358 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:44 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260924 15:27:44.351406 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6411fcadf045d680dbecbb0e303a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:44 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:59612
I20260924 15:27:44.351593 7807 tablet_replica.cc:330] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: stopping tablet replica
I20260924 15:27:44.351735 7807 raft_consensus.cc:2229] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:44.351810 7807 raft_consensus.cc:2258] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:44.351758 7809 tablet_replica.cc:330] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: stopping tablet replica
I20260924 15:27:44.351910 7255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:44.352061 7255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:44.352205 7809 raft_consensus.cc:2229] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:44.352250 7807 ts_tablet_manager.cc:1905] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:44.352432 7809 raft_consensus.cc:2258] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:44.352862 7809 ts_tablet_manager.cc:1905] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:44.353398 7534 tablet_service.cc:1467] Processing CreateTablet for tablet a20a5149bf494fdd9f02649253a2535b (DEFAULT_TABLE table=test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:44.353542 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20a5149bf494fdd9f02649253a2535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:44.354117 7807 ts_tablet_manager.cc:1918] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:44.354205 7807 log.cc:1196] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/0a6411fcadf045d680dbecbb0e303a92
I20260924 15:27:44.354506 7807 ts_tablet_manager.cc:1939] T 0a6411fcadf045d680dbecbb0e303a92 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting consensus metadata
I20260924 15:27:44.354732 7808 tablet_replica.cc:330] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: stopping tablet replica
I20260924 15:27:44.354880 7808 raft_consensus.cc:2229] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:44.354938 7767 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Bootstrap starting.
I20260924 15:27:44.355156 7809 ts_tablet_manager.cc:1918] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:44.355261 7809 log.cc:1196] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/0a6411fcadf045d680dbecbb0e303a92
I20260924 15:27:44.355360 7767 tablet_bootstrap.cc:654] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:44.355544 7809 ts_tablet_manager.cc:1939] T 0a6411fcadf045d680dbecbb0e303a92 P 6022c81c34f84a51931f685c4d19a153: Deleting consensus metadata
I20260924 15:27:44.355858 7239 catalog_manager.cc:4841] TS 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133): tablet 0a6411fcadf045d680dbecbb0e303a92 (table test-table [id=6a881b99d9e245d596fe29ea3bbbf613]) successfully deleted
I20260924 15:27:44.355938 7240 catalog_manager.cc:4841] TS 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635): tablet 0a6411fcadf045d680dbecbb0e303a92 (table test-table [id=6a881b99d9e245d596fe29ea3bbbf613]) successfully deleted
I20260924 15:27:44.356057 7393 tablet_service.cc:1467] Processing CreateTablet for tablet a20a5149bf494fdd9f02649253a2535b (DEFAULT_TABLE table=test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:44.356163 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20a5149bf494fdd9f02649253a2535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:44.356236 7767 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: No bootstrap required, opened a new log
I20260924 15:27:44.356290 7767 ts_tablet_manager.cc:1397] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:44.356483 7767 raft_consensus.cc:348] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.356567 7767 raft_consensus.cc:374] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:44.356592 7767 raft_consensus.cc:729] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Initialized, Role: FOLLOWER
I20260924 15:27:44.356638 7767 consensus_queue.cc:260] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.356750 7767 ts_tablet_manager.cc:1428] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:44.357296 7764 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Bootstrap starting.
I20260924 15:27:44.357548 7673 tablet_service.cc:1467] Processing CreateTablet for tablet a20a5149bf494fdd9f02649253a2535b (DEFAULT_TABLE table=test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:44.357657 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20a5149bf494fdd9f02649253a2535b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:44.357692 7764 tablet_bootstrap.cc:654] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:44.358373 7764 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: No bootstrap required, opened a new log
I20260924 15:27:44.358431 7764 ts_tablet_manager.cc:1397] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:44.358569 7764 raft_consensus.cc:348] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.358639 7764 raft_consensus.cc:374] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:44.358659 7764 raft_consensus.cc:729] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963434a5cd84c1aad3fe16961bf25f4, State: Initialized, Role: FOLLOWER
I20260924 15:27:44.358697 7764 consensus_queue.cc:260] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.358790 7764 ts_tablet_manager.cc:1428] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:44.358990 7763 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Bootstrap starting.
I20260924 15:27:44.359416 7763 tablet_bootstrap.cc:654] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:44.354954 7808 raft_consensus.cc:2258] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:44.360303 7763 tablet_bootstrap.cc:492] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: No bootstrap required, opened a new log
I20260924 15:27:44.360359 7763 ts_tablet_manager.cc:1397] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:44.360383 7808 ts_tablet_manager.cc:1905] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:44.360545 7763 raft_consensus.cc:348] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.360626 7763 raft_consensus.cc:374] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:44.360654 7763 raft_consensus.cc:729] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Initialized, Role: FOLLOWER
I20260924 15:27:44.360695 7763 consensus_queue.cc:260] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.360783 7763 ts_tablet_manager.cc:1428] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:44.362073 7808 ts_tablet_manager.cc:1918] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:44.362157 7808 log.cc:1196] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/0a6411fcadf045d680dbecbb0e303a92
I20260924 15:27:44.362433 7808 ts_tablet_manager.cc:1939] T 0a6411fcadf045d680dbecbb0e303a92 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting consensus metadata
I20260924 15:27:44.362959 7240 catalog_manager.cc:4841] TS f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715): tablet 0a6411fcadf045d680dbecbb0e303a92 (table test-table [id=6a881b99d9e245d596fe29ea3bbbf613]) successfully deleted
I20260924 15:27:44.443385 7798 raft_consensus.cc:482] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:44.443526 7798 raft_consensus.cc:504] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.443780 7798 leader_election.cc:288] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:44.443979 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20a5149bf494fdd9f02649253a2535b" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4" is_pre_election: true
I20260924 15:27:44.444023 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20a5149bf494fdd9f02649253a2535b" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" is_pre_election: true
I20260924 15:27:44.444092 7414 raft_consensus.cc:2454] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:44.444132 7554 raft_consensus.cc:2454] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:44.444262 7625 leader_election.cc:302] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963434a5cd84c1aad3fe16961bf25f4, 6022c81c34f84a51931f685c4d19a153; no voters:
I20260924 15:27:44.444365 7798 raft_consensus.cc:2781] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:44.444402 7798 raft_consensus.cc:482] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:44.444422 7798 raft_consensus.cc:3037] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.445282 7798 raft_consensus.cc:504] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.445425 7798 leader_election.cc:288] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Requested vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:44.445673 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20a5149bf494fdd9f02649253a2535b" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4"
I20260924 15:27:44.445760 7414 raft_consensus.cc:3037] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.445719 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20a5149bf494fdd9f02649253a2535b" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
I20260924 15:27:44.445814 7554 raft_consensus.cc:3037] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:44.446416 7554 raft_consensus.cc:2454] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:44.446549 7414 raft_consensus.cc:2454] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:44.446604 7626 leader_election.cc:302] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:44.446722 7798 raft_consensus.cc:2781] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:44.446771 7798 raft_consensus.cc:686] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Becoming Leader. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Running, Role: LEADER
I20260924 15:27:44.446816 7798 consensus_queue.cc:237] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:44.447465 7254 catalog_manager.cc:5495] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6022c81c34f84a51931f685c4d19a153 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } health_report { overall_health: HEALTHY } } }
I20260924 15:27:44.457827 7414 raft_consensus.cc:1261] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:44.457855 7554 raft_consensus.cc:1261] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:44.458069 7770 consensus_queue.cc:1035] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.458169 7798 consensus_queue.cc:1035] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:44.459316 7785 mvcc.cc:205] Tried to move back new op lower bound from 7332919969617915904 to 7332919969574514688. Current Snapshot: MvccSnapshot[applied={T|T < 7332919969617915904}]
W20260924 15:27:44.465212 7785 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:27:45.982792 7255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:45.982906 7255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:45.983811 7255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Sending DeleteTablet for 3 replicas of tablet a20a5149bf494fdd9f02649253a2535b
I20260924 15:27:45.984140 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet a20a5149bf494fdd9f02649253a2535b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:45 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260924 15:27:45.984155 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet a20a5149bf494fdd9f02649253a2535b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:45 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260924 15:27:45.984175 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet a20a5149bf494fdd9f02649253a2535b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:45 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:59612
I20260924 15:27:45.986372 7837 tablet_replica.cc:330] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: stopping tablet replica
I20260924 15:27:45.986461 7837 raft_consensus.cc:2229] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:45.986509 7837 raft_consensus.cc:2258] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:45.986618 7836 tablet_replica.cc:330] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: stopping tablet replica
I20260924 15:27:45.986692 7836 raft_consensus.cc:2229] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:45.986734 7836 raft_consensus.cc:2258] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:45.986958 7837 ts_tablet_manager.cc:1905] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:45.986982 7836 ts_tablet_manager.cc:1905] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:45.988552 7838 tablet_replica.cc:330] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: stopping tablet replica
I20260924 15:27:45.988564 7837 ts_tablet_manager.cc:1918] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:27:45.988660 7838 raft_consensus.cc:2229] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:45.988725 7838 raft_consensus.cc:2258] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:45.988742 7837 log.cc:1196] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/a20a5149bf494fdd9f02649253a2535b
I20260924 15:27:45.988895 7836 ts_tablet_manager.cc:1918] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:27:45.988960 7836 log.cc:1196] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/a20a5149bf494fdd9f02649253a2535b
I20260924 15:27:45.988979 7838 ts_tablet_manager.cc:1905] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:45.989243 7836 ts_tablet_manager.cc:1939] T a20a5149bf494fdd9f02649253a2535b P 3963434a5cd84c1aad3fe16961bf25f4: Deleting consensus metadata
I20260924 15:27:45.989243 7837 ts_tablet_manager.cc:1939] T a20a5149bf494fdd9f02649253a2535b P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting consensus metadata
I20260924 15:27:45.989574 7239 catalog_manager.cc:4841] TS 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133): tablet a20a5149bf494fdd9f02649253a2535b (table test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]) successfully deleted
I20260924 15:27:45.990295 7240 catalog_manager.cc:4841] TS f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715): tablet a20a5149bf494fdd9f02649253a2535b (table test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]) successfully deleted
I20260924 15:27:45.990613 7838 ts_tablet_manager.cc:1918] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:27:45.990669 7838 log.cc:1196] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/a20a5149bf494fdd9f02649253a2535b
I20260924 15:27:45.990890 7838 ts_tablet_manager.cc:1939] T a20a5149bf494fdd9f02649253a2535b P 6022c81c34f84a51931f685c4d19a153: Deleting consensus metadata
I20260924 15:27:45.991209 7240 catalog_manager.cc:4841] TS 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635): tablet a20a5149bf494fdd9f02649253a2535b (table test-table [id=c1f0989b8a3b478ab015a29c5eb0895c]) successfully deleted
Sep 24 15:27:45 dist-test-slave-4g6f krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:27:46 dist-test-slave-4g6f krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263665, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:46.012810 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.000367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.5.98.254:33889)
0924 15:27:46.000524 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.000526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.000542 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:27:46.000643 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.000644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.000655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.000740 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:46.001307 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.001822 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.002595 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.002803 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.002885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.009897 (+ 7012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:46.009899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:46.009899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:46.009933 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:46.011469 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:46.011957 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:46.011959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:46.011959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:46.012021 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:46.012226 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:46.012227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:46.012228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:46.012349 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:46.012454 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.012604 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.012696 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260924 15:27:46.014276 7255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:46.014446 7255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:46.015825 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (DEFAULT_TABLE table=test-table [id=b96fb105d3884e60805409d86328f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.015851 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (DEFAULT_TABLE table=test-table [id=b96fb105d3884e60805409d86328f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.015970 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.015970 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.015959 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (DEFAULT_TABLE table=test-table [id=b96fb105d3884e60805409d86328f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.016084 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.017113 7847 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Bootstrap starting.
I20260924 15:27:46.017398 7849 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Bootstrap starting.
I20260924 15:27:46.017551 7848 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Bootstrap starting.
I20260924 15:27:46.017726 7847 tablet_bootstrap.cc:654] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.018034 7849 tablet_bootstrap.cc:654] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.018275 7848 tablet_bootstrap.cc:654] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.018404 7847 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: No bootstrap required, opened a new log
I20260924 15:27:46.018481 7847 ts_tablet_manager.cc:1397] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:46.018601 7847 raft_consensus.cc:348] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.018678 7847 raft_consensus.cc:374] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.018711 7847 raft_consensus.cc:729] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.018785 7847 consensus_queue.cc:260] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.018997 7847 ts_tablet_manager.cc:1428] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:46.019083 7848 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: No bootstrap required, opened a new log
I20260924 15:27:46.019135 7848 ts_tablet_manager.cc:1397] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:46.019448 7849 tablet_bootstrap.cc:492] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: No bootstrap required, opened a new log
I20260924 15:27:46.019407 7848 raft_consensus.cc:348] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.019479 7848 raft_consensus.cc:374] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.019500 7848 raft_consensus.cc:729] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963434a5cd84c1aad3fe16961bf25f4, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.019500 7849 ts_tablet_manager.cc:1397] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:46.019567 7848 consensus_queue.cc:260] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.019603 7849 raft_consensus.cc:348] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.019711 7849 raft_consensus.cc:374] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.019740 7849 raft_consensus.cc:729] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.019806 7849 consensus_queue.cc:260] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.019840 7848 ts_tablet_manager.cc:1428] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:46.019902 7849 ts_tablet_manager.cc:1428] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:46.050087 7824 raft_consensus.cc:482] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:46.050179 7824 raft_consensus.cc:504] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.050355 7824 leader_election.cc:288] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:46.050565 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b3fbb3046e4d6e9f2b3fb1fafd3039" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" is_pre_election: true
I20260924 15:27:46.050585 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b3fbb3046e4d6e9f2b3fb1fafd3039" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4" is_pre_election: true
I20260924 15:27:46.050675 7414 raft_consensus.cc:2454] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:46.050680 7554 raft_consensus.cc:2454] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 0.
I20260924 15:27:46.050855 7625 leader_election.cc:302] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963434a5cd84c1aad3fe16961bf25f4, 6022c81c34f84a51931f685c4d19a153; no voters:
I20260924 15:27:46.050995 7824 raft_consensus.cc:2781] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:46.051035 7824 raft_consensus.cc:482] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:46.051050 7824 raft_consensus.cc:3037] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.051797 7824 raft_consensus.cc:504] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.051934 7824 leader_election.cc:288] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Requested vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715)
I20260924 15:27:46.052079 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b3fbb3046e4d6e9f2b3fb1fafd3039" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4"
I20260924 15:27:46.052155 7414 raft_consensus.cc:3037] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.052094 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b3fbb3046e4d6e9f2b3fb1fafd3039" candidate_uuid: "6022c81c34f84a51931f685c4d19a153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b"
I20260924 15:27:46.052193 7554 raft_consensus.cc:3037] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.052845 7554 raft_consensus.cc:2454] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:46.052937 7414 raft_consensus.cc:2454] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6022c81c34f84a51931f685c4d19a153 in term 1.
I20260924 15:27:46.053027 7626 leader_election.cc:302] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:46.053113 7824 raft_consensus.cc:2781] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:46.053162 7824 raft_consensus.cc:686] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Becoming Leader. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Running, Role: LEADER
I20260924 15:27:46.053213 7824 consensus_queue.cc:237] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.053905 7255 catalog_manager.cc:5495] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6022c81c34f84a51931f685c4d19a153 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "6022c81c34f84a51931f685c4d19a153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } health_report { overall_health: HEALTHY } } }
I20260924 15:27:46.068079 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.065644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.5.98.195:42635)
0924 15:27:46.065848 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.065849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.065855 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:27:46.065873 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.065873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.065882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.065954 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:46.066078 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.066580 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.067366 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.067585 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.067659 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.067809 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.067847 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.067907 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.067962 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.067988 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":138,"threads_started":1}
I20260924 15:27:46.070253 7554 raft_consensus.cc:1261] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:46.070310 7414 raft_consensus.cc:1261] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Refusing update from remote peer 6022c81c34f84a51931f685c4d19a153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:46.070454 7824 consensus_queue.cc:1035] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:46.070546 7773 consensus_queue.cc:1035] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:46.080144 7255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:46.080231 7255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:46.080845 7255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Sending DeleteTablet for 3 replicas of tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039
I20260924 15:27:46.081102 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:46 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260924 15:27:46.081141 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:46 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260924 15:27:46.081157 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:46 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:59612
I20260924 15:27:46.081202 7837 tablet_replica.cc:330] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: stopping tablet replica
I20260924 15:27:46.081234 7838 tablet_replica.cc:330] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: stopping tablet replica
I20260924 15:27:46.081236 7836 tablet_replica.cc:330] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: stopping tablet replica
I20260924 15:27:46.081250 7837 raft_consensus.cc:2229] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:46.081274 7838 raft_consensus.cc:2229] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:46.081288 7837 raft_consensus.cc:2258] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:46.081290 7836 raft_consensus.cc:2229] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:46.081322 7838 raft_consensus.cc:2258] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:46.081336 7836 raft_consensus.cc:2258] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:46.081377 7255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:46.081537 7255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:46.081697 7838 ts_tablet_manager.cc:1905] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:46.081753 7837 ts_tablet_manager.cc:1905] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:46.081912 7836 ts_tablet_manager.cc:1905] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:46.083756 7837 ts_tablet_manager.cc:1918] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:46.083774 7836 ts_tablet_manager.cc:1918] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:46.083817 7837 log.cc:1196] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/56b3fbb3046e4d6e9f2b3fb1fafd3039
I20260924 15:27:46.083833 7836 log.cc:1196] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/56b3fbb3046e4d6e9f2b3fb1fafd3039
I20260924 15:27:46.084076 7837 ts_tablet_manager.cc:1939] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P f2cdf979af2d48dd9b93cc2b05819f4b: Deleting consensus metadata
I20260924 15:27:46.084132 7836 ts_tablet_manager.cc:1939] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 3963434a5cd84c1aad3fe16961bf25f4: Deleting consensus metadata
I20260924 15:27:46.084362 7240 catalog_manager.cc:4841] TS f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194:42715): tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (table test-table [id=b96fb105d3884e60805409d86328f26c]) successfully deleted
I20260924 15:27:46.084438 7838 ts_tablet_manager.cc:1918] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:27:46.084424 7239 catalog_manager.cc:4841] TS 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133): tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (table test-table [id=b96fb105d3884e60805409d86328f26c]) successfully deleted
I20260924 15:27:46.084486 7838 log.cc:1196] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/56b3fbb3046e4d6e9f2b3fb1fafd3039
I20260924 15:27:46.084599 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d (DEFAULT_TABLE table=test-table [id=29aeb631514042ada15e0f7b83d5b820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.084690 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3298f7f334664bb1b7252e2e1ecd490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.084697 7838 ts_tablet_manager.cc:1939] T 56b3fbb3046e4d6e9f2b3fb1fafd3039 P 6022c81c34f84a51931f685c4d19a153: Deleting consensus metadata
I20260924 15:27:46.084679 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d (DEFAULT_TABLE table=test-table [id=29aeb631514042ada15e0f7b83d5b820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.084770 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3298f7f334664bb1b7252e2e1ecd490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.085721 7847 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b: Bootstrap starting.
I20260924 15:27:46.085841 7848 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: Bootstrap starting.
I20260924 15:27:46.085938 7240 catalog_manager.cc:4841] TS 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635): tablet 56b3fbb3046e4d6e9f2b3fb1fafd3039 (table test-table [id=b96fb105d3884e60805409d86328f26c]) successfully deleted
I20260924 15:27:46.086037 7673 tablet_service.cc:1467] Processing CreateTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d (DEFAULT_TABLE table=test-table [id=29aeb631514042ada15e0f7b83d5b820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:46.086133 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3298f7f334664bb1b7252e2e1ecd490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:46.086229 7848 tablet_bootstrap.cc:654] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.086294 7847 tablet_bootstrap.cc:654] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.086824 7848 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: No bootstrap required, opened a new log
I20260924 15:27:46.086884 7848 ts_tablet_manager.cc:1397] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:46.086956 7847 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b: No bootstrap required, opened a new log
I20260924 15:27:46.087016 7847 ts_tablet_manager.cc:1397] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:46.087044 7848 raft_consensus.cc:348] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.087114 7848 raft_consensus.cc:374] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.087138 7848 raft_consensus.cc:729] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963434a5cd84c1aad3fe16961bf25f4, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.087177 7848 consensus_queue.cc:260] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.087169 7847 raft_consensus.cc:348] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.087239 7847 raft_consensus.cc:374] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.087267 7848 ts_tablet_manager.cc:1428] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:46.087270 7847 raft_consensus.cc:729] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.087320 7847 consensus_queue.cc:260] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.087417 7847 ts_tablet_manager.cc:1428] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:46.087440 7849 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Bootstrap starting.
I20260924 15:27:46.087751 7849 tablet_bootstrap.cc:654] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:46.088327 7849 tablet_bootstrap.cc:492] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: No bootstrap required, opened a new log
I20260924 15:27:46.088366 7849 ts_tablet_manager.cc:1397] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:46.088444 7849 raft_consensus.cc:348] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.088490 7849 raft_consensus.cc:374] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:46.088503 7849 raft_consensus.cc:729] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6022c81c34f84a51931f685c4d19a153, State: Initialized, Role: FOLLOWER
I20260924 15:27:46.088562 7849 consensus_queue.cc:260] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.088678 7849 ts_tablet_manager.cc:1428] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:46.162019 7853 raft_consensus.cc:482] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:46.162151 7853 raft_consensus.cc:504] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.162600 7853 leader_election.cc:288] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635)
I20260924 15:27:46.166998 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.162873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59041 (local address 127.5.98.195:42635)
0924 15:27:46.162923 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.162925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.163070 (+ 145us) server_negotiation.cc:407] Connection header received
0924 15:27:46.163143 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.163145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.163155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.163211 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:46.163522 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.164341 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.165976 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.166541 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.166634 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.166643 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.166877 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.166916 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:27:46.166991 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.162832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:55097 (local address 127.5.98.193:36133)
0924 15:27:46.162988 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.162992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.163006 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:27:46.163111 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.163112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.163124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.163209 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:46.163588 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.164361 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.165976 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.166566 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.166635 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.166643 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.166877 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.166917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260924 15:27:46.167243 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.162802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.5.98.194:59041)
0924 15:27:46.162984 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:46.162993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:46.163067 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:46.163230 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:46.163230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:46.163234 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:46.163341 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.163343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.164439 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.164441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.165868 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.165871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.166667 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.166669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.166766 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.166773 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:27:46.166856 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:27:46.166916 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":85,"thread_start_us":99,"threads_started":1}
I20260924 15:27:46.167336 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.162736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36133 (local address 127.5.98.194:55097)
0924 15:27:46.162940 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:46.162960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:46.163020 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:46.163229 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:46.163230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:46.163233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:46.163507 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.163510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.164459 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.164460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.165868 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.165871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.166669 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.166669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.166785 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.166793 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:27:46.166856 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:27:46.166916 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":108,"thread_start_us":77,"threads_started":1}
I20260924 15:27:46.167549 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3298f7f334664bb1b7252e2e1ecd490d" candidate_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6022c81c34f84a51931f685c4d19a153" is_pre_election: true
I20260924 15:27:46.167693 7694 raft_consensus.cc:2454] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2cdf979af2d48dd9b93cc2b05819f4b in term 0.
I20260924 15:27:46.167920 7486 leader_election.cc:302] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6022c81c34f84a51931f685c4d19a153, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:46.167906 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3298f7f334664bb1b7252e2e1ecd490d" candidate_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4" is_pre_election: true
I20260924 15:27:46.167994 7414 raft_consensus.cc:2454] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2cdf979af2d48dd9b93cc2b05819f4b in term 0.
I20260924 15:27:46.168080 7853 raft_consensus.cc:2781] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:46.168126 7853 raft_consensus.cc:482] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:46.168147 7853 raft_consensus.cc:3037] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.168851 7853 raft_consensus.cc:504] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.168987 7853 leader_election.cc:288] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [CANDIDATE]: Term 1 election: Requested vote from peers 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133), 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635)
I20260924 15:27:46.169147 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3298f7f334664bb1b7252e2e1ecd490d" candidate_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6022c81c34f84a51931f685c4d19a153"
I20260924 15:27:46.169159 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3298f7f334664bb1b7252e2e1ecd490d" candidate_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963434a5cd84c1aad3fe16961bf25f4"
I20260924 15:27:46.169227 7414 raft_consensus.cc:3037] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.169229 7694 raft_consensus.cc:3037] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:46.169862 7694 raft_consensus.cc:2454] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2cdf979af2d48dd9b93cc2b05819f4b in term 1.
I20260924 15:27:46.169867 7414 raft_consensus.cc:2454] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2cdf979af2d48dd9b93cc2b05819f4b in term 1.
I20260924 15:27:46.170078 7485 leader_election.cc:302] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963434a5cd84c1aad3fe16961bf25f4, f2cdf979af2d48dd9b93cc2b05819f4b; no voters:
I20260924 15:27:46.170185 7853 raft_consensus.cc:2781] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:46.170341 7853 raft_consensus.cc:686] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [term 1 LEADER]: Becoming Leader. State: Replica: f2cdf979af2d48dd9b93cc2b05819f4b, State: Running, Role: LEADER
I20260924 15:27:46.170423 7853 consensus_queue.cc:237] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } }
I20260924 15:27:46.171146 7255 catalog_manager.cc:5495] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to f2cdf979af2d48dd9b93cc2b05819f4b (127.5.98.194). New cstate: current_term: 1 leader_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2cdf979af2d48dd9b93cc2b05819f4b" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 } health_report { overall_health: UNKNOWN } } }
I20260924 15:27:46.182125 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.179911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.5.98.194:42715)
0924 15:27:46.180068 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.180071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.180078 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:46.180119 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.180120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.180136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.180207 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:46.180300 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.180760 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.181435 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.181639 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.181691 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.181804 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.181905 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.181952 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.181987 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.182002 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260924 15:27:46.187013 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.184600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:48063 (local address 127.5.98.195:42635)
0924 15:27:46.184637 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.184639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.184834 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:27:46.184860 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.184861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.184870 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.184923 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:46.185036 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.185471 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.186312 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.186500 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.186570 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.186778 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.186813 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.186865 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.186942 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.186961 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:46.187049 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.184457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42635 (local address 127.5.98.194:48063)
0924 15:27:46.184736 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:46.184750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:46.184814 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:46.184945 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:46.184946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:46.184947 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:46.184994 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.184996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.185560 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.185561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.186205 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.186208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.186608 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.186609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.186662 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.186677 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.186886 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.186887 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:27:46.186938 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:27:46.186977 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":210,"threads_started":1}
I20260924 15:27:46.189394 7414 raft_consensus.cc:1261] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4 [term 1 FOLLOWER]: Refusing update from remote peer f2cdf979af2d48dd9b93cc2b05819f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:46.189443 7694 raft_consensus.cc:1261] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Refusing update from remote peer f2cdf979af2d48dd9b93cc2b05819f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:46.189695 7867 consensus_queue.cc:1035] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963434a5cd84c1aad3fe16961bf25f4" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:46.189867 7853 consensus_queue.cc:1035] T 3298f7f334664bb1b7252e2e1ecd490d P f2cdf979af2d48dd9b93cc2b05819f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6022c81c34f84a51931f685c4d19a153" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:46.197014 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.194598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:47007 (local address 127.5.98.194:42715)
0924 15:27:46.194643 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:46.194645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:46.194817 (+ 172us) server_negotiation.cc:407] Connection header received
0924 15:27:46.194880 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:46.194882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:46.194892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:46.194946 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:46.195060 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.195546 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.196335 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.196570 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.196646 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.196788 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.196828 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.196883 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:46.196940 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:27:46.196960 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:27:46.197085 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:46.194553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42715 (local address 127.5.98.195:47007)
0924 15:27:46.194756 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:46.194765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:46.194830 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:46.194949 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:46.194949 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:46.194951 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:27:46.195002 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.195004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.195631 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.195632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.196234 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:46.196236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:46.196646 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:46.196647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:46.196708 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:46.196713 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:27:46.196892 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:27:46.196894 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:27:46.196925 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 15:27:46.197003 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
W20260924 15:27:46.197504 7821 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:27:47.721060 7254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:47.721168 7254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:47.721962 7254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fad5621f60794d8ea3d4a03a8ee6597b: Sending DeleteTablet for 3 replicas of tablet 3298f7f334664bb1b7252e2e1ecd490d
I20260924 15:27:47.722204 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:47 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:53954
I20260924 15:27:47.722271 7673 tablet_service.cc:1514] Processing DeleteTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:47 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:59612
I20260924 15:27:47.722293 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 3298f7f334664bb1b7252e2e1ecd490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:47 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:54302
I20260924 15:27:47.722389 7900 tablet_replica.cc:330] T 3298f7f334664bb1b7252e2e1ecd490d P 3963434a5cd84c1aad3fe16961bf25f4: stopping tablet replica
I20260924 15:27:47.722373 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 7328
W20260924 15:27:47.727897 7239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:36133 (error 108)
I20260924 15:27:47.728530 7902 tablet_replica.cc:330] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: stopping tablet replica
I20260924 15:27:47.728608 7902 raft_consensus.cc:2229] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:47.728660 7902 raft_consensus.cc:2258] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:47.728693 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 7469
I20260924 15:27:47.728720 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:47.728475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36133 (local address 127.0.0.1:53968)
0924 15:27:47.728621 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:47.728675 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20260924 15:27:47.728811 7239 catalog_manager.cc:4584] TS 3963434a5cd84c1aad3fe16961bf25f4 (127.5.98.193:36133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3298f7f334664bb1b7252e2e1ecd490d: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36133: connect: Connection refused (error 111)
I20260924 15:27:47.730401 7902 ts_tablet_manager.cc:1905] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:47.732298 7902 ts_tablet_manager.cc:1918] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:27:47.732369 7902 log.cc:1196] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/3298f7f334664bb1b7252e2e1ecd490d
I20260924 15:27:47.732626 7902 ts_tablet_manager.cc:1939] T 3298f7f334664bb1b7252e2e1ecd490d P 6022c81c34f84a51931f685c4d19a153: Deleting consensus metadata
I20260924 15:27:47.733002 7240 catalog_manager.cc:4841] TS 6022c81c34f84a51931f685c4d19a153 (127.5.98.195:42635): tablet 3298f7f334664bb1b7252e2e1ecd490d (table test-table [id=29aeb631514042ada15e0f7b83d5b820]) successfully deleted
I20260924 15:27:47.736527 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:47.736392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42715 (local address 127.0.0.1:54326)
0924 15:27:47.736443 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:47.736469 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.194:42715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:27:47.736647 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 7609
I20260924 15:27:47.744109 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 7223
2026-09-24T15:27:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5787 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:47 dist-test-slave-4g6f krb5kdc[7907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:47 dist-test-slave-4g6f krb5kdc[7907](info): set up 2 sockets
Sep 24 15:27:47 dist-test-slave-4g6f krb5kdc[7907](info): commencing operation
krb5kdc: starting...
W20260924 15:27:48.777446 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:48 dist-test-slave-4g6f krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:48.894807 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:45153
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:45153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:48.976701 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:48.976928 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:48.976965 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:48.978102 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:48.978171 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:48.978194 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:48.978217 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:48.978245 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:48.979717 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:45153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:45153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:48.979990 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:48.980209 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:48.985025 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:48.985070 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:48.985225 7920 server_base.cc:1034] running on GCE node
W20260924 15:27:48.985093 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:48.985553 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:48.985849 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:48.987063 7920 hybrid_clock.cc:648] HybridClock initialized: now 1790263668987047 us; error 31 us; skew 500 ppm
Sep 24 15:27:48 dist-test-slave-4g6f krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263668, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:48.989591 7920 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:48.990054 7920 webserver.cc:460] Webserver started at http://127.5.98.254:32877/ using document root <none> and password file <none>
I20260924 15:27:48.990283 7920 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:48.990329 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:48.990480 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:48.991556 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "2740be2e35f242ad9aad8300cf46270f"
format_stamp: "Formatted at 2026-09-24 15:27:48 on dist-test-slave-4g6f"
server_key: "66889c071183a62f73ce9eb5b9de01d5"
server_key_iv: "58d36809382f88cb92c35d70ca291703"
server_key_version: "encryptionkey@0"
I20260924 15:27:48.992065 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "2740be2e35f242ad9aad8300cf46270f"
format_stamp: "Formatted at 2026-09-24 15:27:48 on dist-test-slave-4g6f"
server_key: "66889c071183a62f73ce9eb5b9de01d5"
server_key_iv: "58d36809382f88cb92c35d70ca291703"
server_key_version: "encryptionkey@0"
I20260924 15:27:48.993412 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:48.994186 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:48.994362 7920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:48.994449 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "2740be2e35f242ad9aad8300cf46270f"
format_stamp: "Formatted at 2026-09-24 15:27:48 on dist-test-slave-4g6f"
server_key: "66889c071183a62f73ce9eb5b9de01d5"
server_key_iv: "58d36809382f88cb92c35d70ca291703"
server_key_version: "encryptionkey@0"
I20260924 15:27:48.994517 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:49.011960 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:49.014160 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:49.014268 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:49.018180 7920 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:45153
I20260924 15:27:49.018260 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:45153 every 8 connection(s)
I20260924 15:27:49.018616 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:49.019286 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 7920
I20260924 15:27:49.019388 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:49.019579 5515 external_mini_cluster.cc:1428] Setting key 4ca2b62d3ba98c0559e4b49f93f42bff
I20260924 15:27:49.019611 7988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:49.023061 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: Bootstrap starting.
I20260924 15:27:49.023754 7988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:49.024091 7988 log.cc:824] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:49.025038 7988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: No bootstrap required, opened a new log
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:49.027249 7988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } }
I20260924 15:27:49.027381 7988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:49.027405 7988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2740be2e35f242ad9aad8300cf46270f, State: Initialized, Role: FOLLOWER
I20260924 15:27:49.027545 7988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } }
I20260924 15:27:49.027626 7988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:49.027647 7988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:49.027737 7988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:49.028410 7988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } }
I20260924 15:27:49.028537 7988 leader_election.cc:302] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2740be2e35f242ad9aad8300cf46270f; no voters:
I20260924 15:27:49.028734 7988 leader_election.cc:288] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:49.028762 7993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:49.028872 7993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [term 1 LEADER]: Becoming Leader. State: Replica: 2740be2e35f242ad9aad8300cf46270f, State: Running, Role: LEADER
I20260924 15:27:49.028971 7993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } }
I20260924 15:27:49.028996 7988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:49.030679 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2740be2e35f242ad9aad8300cf46270f. Latest consensus state: current_term: 1 leader_uuid: "2740be2e35f242ad9aad8300cf46270f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } } }
I20260924 15:27:49.031014 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2740be2e35f242ad9aad8300cf46270f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2740be2e35f242ad9aad8300cf46270f" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45153 } } }
I20260924 15:27:49.031252 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:49.031091 8007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 15:27:49.031239 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:49.031062 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.020133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.5.98.254:45153)
0924 15:27:49.020340 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:49.020351 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:49.020383 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:27:49.020838 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:49.020847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:49.020922 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:49.021081 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:49.021689 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.023070 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.024015 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.024738 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.024850 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.027539 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:49.027542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:49.027543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:49.027553 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:49.029416 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.029854 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.029857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.029857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.029955 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.030165 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.030166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.030166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.030402 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:49.030579 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:49.030666 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:27:49.030802 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
W20260924 15:27:49.031378 8007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:49.031778 8010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:49.031978 8010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:49.033413 8010 catalog_manager.cc:1347] Generated new cluster ID: 5c693f3740cc4d59b0c088286e194199
I20260924 15:27:49.033468 8010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:49.052553 8010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:49.053297 8010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:49.056946 8010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2740be2e35f242ad9aad8300cf46270f: Generated new TSK 0
I20260924 15:27:49.057143 8010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:49.096920 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:49.201674 8014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:49.201879 8014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:49.201901 8014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:49.203125 8014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:49.203168 8014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:49.203222 8014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:49.204650 8014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:49.204906 8014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:49.205152 8014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:49.210142 8020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:49.210150 8022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:49.210291 8014 server_base.cc:1034] running on GCE node
W20260924 15:27:49.210173 8019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:49.210580 8014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:49.210821 8014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:49.211980 8014 hybrid_clock.cc:648] HybridClock initialized: now 1790263669211962 us; error 51 us; skew 500 ppm
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:49.214588 8014 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:49.214954 8014 webserver.cc:460] Webserver started at http://127.5.98.193:33167/ using document root <none> and password file <none>
I20260924 15:27:49.215164 8014 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:49.215207 8014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:49.215301 8014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:49.216356 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "462dec7668bb4f169f23c8224d0ca563"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "58aef444e792f86d208c960ed01e6cd4"
server_key_iv: "38aea42a7378014fc2bc167c6a509c63"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.216658 8014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "462dec7668bb4f169f23c8224d0ca563"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "58aef444e792f86d208c960ed01e6cd4"
server_key_iv: "38aea42a7378014fc2bc167c6a509c63"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.217875 8014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:49.218586 8029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.218770 8014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:49.218863 8014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "462dec7668bb4f169f23c8224d0ca563"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "58aef444e792f86d208c960ed01e6cd4"
server_key_iv: "38aea42a7378014fc2bc167c6a509c63"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.218924 8014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:49.241300 8014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:49.243804 8014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:49.243937 8014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:49.244145 8014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:49.244493 8014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:49.244532 8014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.244555 8014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:49.244568 8014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.250284 8014 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:38431
I20260924 15:27:49.250363 8142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:38431 every 8 connection(s)
I20260924 15:27:49.250698 8014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:49.252897 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8014
I20260924 15:27:49.252982 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:49.253196 5515 external_mini_cluster.cc:1428] Setting key 7284de6ecdb8d2470aa6bc24fa3446fe
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:49.261660 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.251468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:38697 (local address 127.5.98.254:45153)
0924 15:27:49.251531 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:49.251533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:49.251817 (+ 284us) server_negotiation.cc:407] Connection header received
0924 15:27:49.252337 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:49.252339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:49.252351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:49.252413 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:49.253504 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.254249 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.255530 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.255797 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.255861 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.258536 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:49.258537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:49.258538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:49.258544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:49.259996 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.260518 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.260519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.260519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.260568 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.260907 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.260908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.260908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.261049 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:49.261114 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:49.261529 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 15:27:49.261597 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:27:49.261914 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.251472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:45153 (local address 127.5.98.193:38697)
0924 15:27:49.251628 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:49.251713 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:49.252211 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:49.252478 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:49.252482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:49.253109 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:49.253374 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.253382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.254362 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.254362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.255434 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.255437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.255881 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.255882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.256044 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.256667 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:49.256676 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:49.258433 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:49.260080 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.260089 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.260091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.260430 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.260637 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.260638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.260639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.260791 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.261132 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:49.261138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:49.261309 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 15:27:49.261459 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 15:27:49.261638 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
I20260924 15:27:49.262571 8143 heartbeater.cc:347] Connected to a master server at 127.5.98.254:45153
I20260924 15:27:49.262694 8143 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:49.262873 8143 heartbeater.cc:511] Master 127.5.98.254:45153 requested a full tablet report, sending...
I20260924 15:27:49.263396 7952 ts_manager.cc:194] Registered new tserver with Master: 462dec7668bb4f169f23c8224d0ca563 (127.5.98.193:38431)
I20260924 15:27:49.264142 7952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:38697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:49.278950 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:49.359789 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:49.360016 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:49.360054 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:49.361191 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:49.361263 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:49.361310 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:49.362735 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:49.362995 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:49.363227 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:49.368172 8153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:49.368237 8154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:49.368160 8156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:49.368295 8148 server_base.cc:1034] running on GCE node
I20260924 15:27:49.368605 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:49.368870 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:49.370034 8148 hybrid_clock.cc:648] HybridClock initialized: now 1790263669370012 us; error 39 us; skew 500 ppm
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:49.372573 8148 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:49.373000 8148 webserver.cc:460] Webserver started at http://127.5.98.194:33393/ using document root <none> and password file <none>
I20260924 15:27:49.373251 8148 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:49.373339 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:49.373488 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:49.374526 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "7f05b66cbb3641e29d5df96614649dfe"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "90c291b677defd75afed175e1678d538"
server_key_iv: "24c5a6cad68fafe8961416a5b30a5722"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.374879 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "7f05b66cbb3641e29d5df96614649dfe"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "90c291b677defd75afed175e1678d538"
server_key_iv: "24c5a6cad68fafe8961416a5b30a5722"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.376256 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 15:27:49.377018 8163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.377218 8148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:49.377311 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "7f05b66cbb3641e29d5df96614649dfe"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "90c291b677defd75afed175e1678d538"
server_key_iv: "24c5a6cad68fafe8961416a5b30a5722"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.377384 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:49.392400 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:49.394469 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:49.394651 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:49.394896 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:49.395241 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:49.395296 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.395330 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:49.395357 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.401397 8148 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:33061
I20260924 15:27:49.401472 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:33061 every 8 connection(s)
I20260924 15:27:49.401813 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:49.403900 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8148
I20260924 15:27:49.404011 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:49.404196 5515 external_mini_cluster.cc:1428] Setting key bae8bb9c5df4d75f85c73d743c52ff12
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:49.412232 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.402568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:42199 (local address 127.5.98.254:45153)
0924 15:27:49.402611 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:49.402616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:49.402806 (+ 190us) server_negotiation.cc:407] Connection header received
0924 15:27:49.403301 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:49.403302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:49.403313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:49.403366 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:49.404621 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.405220 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.406698 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.406898 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.406987 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.409429 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:49.409431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:49.409431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:49.409439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:49.410809 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.411229 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.411230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.411230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.411293 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.411552 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.411552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.411553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.411714 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:49.411773 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:49.412063 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 15:27:49.412141 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:27:49.412431 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.402551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:45153 (local address 127.5.98.194:42199)
0924 15:27:49.402696 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:49.402733 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:49.403210 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:49.403420 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:49.403424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:49.404214 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:49.404518 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.404527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.405316 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.405318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.406604 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.406607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.407005 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.407006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.407128 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.407732 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:49.407740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:49.409316 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:49.410909 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.410916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.410918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.411148 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.411367 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.411368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.411368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.411483 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.411776 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:49.411782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:49.411921 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 15:27:49.412027 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:27:49.412189 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260924 15:27:49.413059 8277 heartbeater.cc:347] Connected to a master server at 127.5.98.254:45153
I20260924 15:27:49.413166 8277 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:49.413383 8277 heartbeater.cc:511] Master 127.5.98.254:45153 requested a full tablet report, sending...
I20260924 15:27:49.413856 7952 ts_manager.cc:194] Registered new tserver with Master: 7f05b66cbb3641e29d5df96614649dfe (127.5.98.194:33061)
I20260924 15:27:49.414322 7952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:42199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:49.428656 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:49.510819 8282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:49.511049 8282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:49.511087 8282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:49.512317 8282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:49.512377 8282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:49.512422 8282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:49.513939 8282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:45153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:49.514204 8282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:49.514433 8282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:49.519341 8288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:49.519331 8287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:49.519403 8282 server_base.cc:1034] running on GCE node
W20260924 15:27:49.519520 8290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:49.519783 8282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:49.520028 8282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:49.521185 8282 hybrid_clock.cc:648] HybridClock initialized: now 1790263669521171 us; error 29 us; skew 500 ppm
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:49.523780 8282 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:49.524194 8282 webserver.cc:460] Webserver started at http://127.5.98.195:38955/ using document root <none> and password file <none>
I20260924 15:27:49.524430 8282 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:49.524507 8282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:49.524664 8282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:49.525730 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "a9cb861d74a84181b2736a83b4e50527"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "bfd49c7fb6bd68cf043d97b2ae96e7cf"
server_key_iv: "8642e2750975ee923d767102f5cccd48"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.526108 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "a9cb861d74a84181b2736a83b4e50527"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "bfd49c7fb6bd68cf043d97b2ae96e7cf"
server_key_iv: "8642e2750975ee923d767102f5cccd48"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.527428 8282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:49.528304 8297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.528535 8282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:49.528635 8282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "a9cb861d74a84181b2736a83b4e50527"
format_stamp: "Formatted at 2026-09-24 15:27:49 on dist-test-slave-4g6f"
server_key: "bfd49c7fb6bd68cf043d97b2ae96e7cf"
server_key_iv: "8642e2750975ee923d767102f5cccd48"
server_key_version: "encryptionkey@0"
I20260924 15:27:49.528723 8282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:49.544485 8282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:49.546360 8282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:49.546523 8282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:49.546742 8282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:49.547047 8282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:49.547102 8282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.547142 8282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:49.547170 8282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:49.552925 8282 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:34165
I20260924 15:27:49.552966 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:34165 every 8 connection(s)
I20260924 15:27:49.553289 8282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[7907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:49.562490 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.553979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:57705 (local address 127.5.98.254:45153)
0924 15:27:49.554030 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:49.554032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:49.554251 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:27:49.554756 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:49.554767 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:49.554777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:49.554841 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:49.555865 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.556362 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.557487 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.557696 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.557777 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.559923 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:49.559925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:49.559925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:49.559931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:49.561278 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.561654 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.561655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.561655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.561703 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:49.561969 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:49.561970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:49.561970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:49.562085 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:49.562138 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:49.562368 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 15:27:49.562427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:49.562887 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:49.553963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:45153 (local address 127.5.98.195:57705)
0924 15:27:49.554133 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:49.554168 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:49.554652 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:49.554942 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:49.554946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:49.555516 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:49.555778 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.555784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.556469 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.556470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.557384 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:49.557387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:49.557784 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:49.557785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:49.557915 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:49.558403 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:49.558412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:49.559840 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:49.561362 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.561368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.561370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.561585 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.561786 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:49.561787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:49.561788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:49.561903 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:49.562148 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:49.562152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:49.562261 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:27:49.562346 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:27:49.562463 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 15:27:49.563374 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8282
I20260924 15:27:49.563418 8411 heartbeater.cc:347] Connected to a master server at 127.5.98.254:45153
I20260924 15:27:49.563481 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:49.563508 8411 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:49.563709 5515 external_mini_cluster.cc:1428] Setting key 95feb6559c9742e52e17bd9884bccde5
I20260924 15:27:49.563719 8411 heartbeater.cc:511] Master 127.5.98.254:45153 requested a full tablet report, sending...
I20260924 15:27:49.564059 7952 ts_manager.cc:194] Registered new tserver with Master: a9cb861d74a84181b2736a83b4e50527 (127.5.98.195:34165)
I20260924 15:27:49.564422 7952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:57705
I20260924 15:27:49.564541 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:27:49.576479 7991 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:49.575441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.5.98.254:45153)
0924 15:27:49.575520 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:49.575523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:49.575627 (+ 104us) server_negotiation.cc:407] Connection header received
0924 15:27:49.575752 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:49.575753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:49.575771 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:49.575835 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:49.576387 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49544: BlockingRecv error: recv got EOF from 127.0.0.1:49544 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 15:27:49.577297 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8014
I20260924 15:27:49.582356 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8148
I20260924 15:27:49.586889 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8282
I20260924 15:27:49.591096 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 7920
2026-09-24T15:27:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1845 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[8423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[8423](info): set up 2 sockets
Sep 24 15:27:49 dist-test-slave-4g6f krb5kdc[8423](info): commencing operation
krb5kdc: starting...
W20260924 15:27:50.628803 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:50 dist-test-slave-4g6f krb5kdc[8423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:50.746500 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:45609
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:45609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:50.840188 8436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:50.840391 8436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:50.840412 8436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:50.841491 8436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:50.841523 8436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:50.841537 8436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:50.841550 8436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:50.841562 8436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:50.842984 8436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:45609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:50.843171 8436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:50.843333 8436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:50.847990 8441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:50.847998 8444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:50.848053 8442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:50.848186 8436 server_base.cc:1034] running on GCE node
I20260924 15:27:50.848510 8436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:50.848749 8436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:50.849887 8436 hybrid_clock.cc:648] HybridClock initialized: now 1790263670849872 us; error 33 us; skew 500 ppm
Sep 24 15:27:50 dist-test-slave-4g6f krb5kdc[8423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263670, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:50.852259 8436 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:50.852735 8436 webserver.cc:460] Webserver started at http://127.5.98.254:33001/ using document root <none> and password file <none>
I20260924 15:27:50.852968 8436 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:50.853036 8436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:50.853184 8436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:50.854195 8436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "7a289142137b4f2398d1efde7ef77c2e"
format_stamp: "Formatted at 2026-09-24 15:27:50 on dist-test-slave-4g6f"
server_key: "85bb3ce6c2a4f9cbe2ba51061533c250"
server_key_iv: "0b8db42359d861905057eece275a59cc"
server_key_version: "encryptionkey@0"
I20260924 15:27:50.854557 8436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "7a289142137b4f2398d1efde7ef77c2e"
format_stamp: "Formatted at 2026-09-24 15:27:50 on dist-test-slave-4g6f"
server_key: "85bb3ce6c2a4f9cbe2ba51061533c250"
server_key_iv: "0b8db42359d861905057eece275a59cc"
server_key_version: "encryptionkey@0"
I20260924 15:27:50.855880 8436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:50.856560 8451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:50.856742 8436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:50.856846 8436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "7a289142137b4f2398d1efde7ef77c2e"
format_stamp: "Formatted at 2026-09-24 15:27:50 on dist-test-slave-4g6f"
server_key: "85bb3ce6c2a4f9cbe2ba51061533c250"
server_key_iv: "0b8db42359d861905057eece275a59cc"
server_key_version: "encryptionkey@0"
I20260924 15:27:50.856920 8436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:50.874639 8436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:50.876500 8436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:50.876672 8436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:50.880645 8436 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:45609
I20260924 15:27:50.880676 8503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:45609 every 8 connection(s)
I20260924 15:27:50.881029 8436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:50.881853 8504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:50.882061 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8436
I20260924 15:27:50.882160 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:50.882345 5515 external_mini_cluster.cc:1428] Setting key af9116cce88ed3e1c8907b2c3f19e87a
I20260924 15:27:50.885185 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e: Bootstrap starting.
I20260924 15:27:50.885840 8504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:50.886152 8504 log.cc:824] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:50.886829 8504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e: No bootstrap required, opened a new log
I20260924 15:27:50.888893 8504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } }
I20260924 15:27:50.889066 8504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:50.889101 8504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a289142137b4f2398d1efde7ef77c2e, State: Initialized, Role: FOLLOWER
Sep 24 15:27:50 dist-test-slave-4g6f krb5kdc[8423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:50.889235 8504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } }
I20260924 15:27:50.889322 8504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:50.889353 8504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:50.889421 8504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:50.890051 8504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } }
I20260924 15:27:50.890166 8504 leader_election.cc:302] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a289142137b4f2398d1efde7ef77c2e; no voters:
I20260924 15:27:50.890364 8504 leader_election.cc:288] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:50.890422 8509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:50.890559 8509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [term 1 LEADER]: Becoming Leader. State: Replica: 7a289142137b4f2398d1efde7ef77c2e, State: Running, Role: LEADER
I20260924 15:27:50.890648 8504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:50.890661 8509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } }
I20260924 15:27:50.890982 8510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a289142137b4f2398d1efde7ef77c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } } }
I20260924 15:27:50.891076 8510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:50.890998 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a289142137b4f2398d1efde7ef77c2e. Latest consensus state: current_term: 1 leader_uuid: "7a289142137b4f2398d1efde7ef77c2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a289142137b4f2398d1efde7ef77c2e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 45609 } } }
I20260924 15:27:50.891258 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:50.891808 8515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:50.892381 8515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:50.893956 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:50.882892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.5.98.254:45609)
0924 15:27:50.883092 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:50.883107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:50.883132 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:27:50.883564 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:50.883571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:50.883615 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:50.883782 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:50.884595 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:50.885567 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:50.886600 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:50.887057 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:50.887197 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:50.889708 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:50.889712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:50.889712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:50.889721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:50.892274 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:50.892700 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:50.892703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:50.892703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:50.892806 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:50.893030 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:50.893031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:50.893031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:50.893341 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:50.893485 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:50.893629 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 15:27:50.893762 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260924 15:27:50.894305 8515 catalog_manager.cc:1347] Generated new cluster ID: f84b473a02834d74917182f4cd12e7a8
I20260924 15:27:50.894366 8515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:50.894850 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:50.921018 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:50.916759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.5.98.254:45609)
0924 15:27:50.916801 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:50.916803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:50.916943 (+ 140us) server_negotiation.cc:407] Connection header received
0924 15:27:50.917001 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:50.917002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:50.917012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:50.917080 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:50.917605 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:50.918331 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:50.919381 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:50.919708 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:50.919769 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:50.920498 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:50.920499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:50.920500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:50.920728 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:50.920749 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:50.920755 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:50.920810 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:50.920906 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:27:50.920930 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:50.944325 8515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:50.944988 8515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:50.947683 8515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a289142137b4f2398d1efde7ef77c2e: Generated new TSK 0
I20260924 15:27:50.947865 8515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:50.954649 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8436
2026-09-24T15:27:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1361 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:51 dist-test-slave-4g6f krb5kdc[8537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:51 dist-test-slave-4g6f krb5kdc[8537](info): set up 2 sockets
Sep 24 15:27:51 dist-test-slave-4g6f krb5kdc[8537](info): commencing operation
krb5kdc: starting...
W20260924 15:27:51.997246 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:52.113756 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43081
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:52.194326 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:52.194583 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:52.194618 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:52.195864 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:52.195930 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:52.195955 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:52.195986 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:52.196015 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:52.197526 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:52.197808 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:52.198051 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:52.202981 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.202989 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.203001 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:52.203191 8550 server_base.cc:1034] running on GCE node
I20260924 15:27:52.203500 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:52.203876 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:52.205039 8550 hybrid_clock.cc:648] HybridClock initialized: now 1790263672205034 us; error 36 us; skew 500 ppm
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:52.214695 8550 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:52.215189 8550 webserver.cc:460] Webserver started at http://127.5.98.254:35793/ using document root <none> and password file <none>
I20260924 15:27:52.215435 8550 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:52.215520 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:52.215723 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:52.216766 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "2cbfb64baacc4003a110f83ee9208268"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fe9ebc7dca84343beedfe3c044c15318"
server_key_iv: "f3c068bcb8a57170d21780228bc63c05"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.217144 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "2cbfb64baacc4003a110f83ee9208268"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fe9ebc7dca84343beedfe3c044c15318"
server_key_iv: "f3c068bcb8a57170d21780228bc63c05"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.218535 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:52.219257 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.219427 8550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.219534 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "2cbfb64baacc4003a110f83ee9208268"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fe9ebc7dca84343beedfe3c044c15318"
server_key_iv: "f3c068bcb8a57170d21780228bc63c05"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.219611 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:52.242715 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:52.244870 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:52.245062 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:52.248871 8550 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43081
I20260924 15:27:52.248911 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43081 every 8 connection(s)
I20260924 15:27:52.249291 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:52.250272 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:52.252586 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268: Bootstrap starting.
I20260924 15:27:52.253101 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:52.253328 8618 log.cc:824] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:52.253986 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268: No bootstrap required, opened a new log
I20260924 15:27:52.255545 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } }
I20260924 15:27:52.255703 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:52.255728 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbfb64baacc4003a110f83ee9208268, State: Initialized, Role: FOLLOWER
I20260924 15:27:52.255865 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } }
I20260924 15:27:52.255949 8618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:52.255971 8618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:52.256001 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:52.256778 8618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } }
I20260924 15:27:52.256865 8618 leader_election.cc:302] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cbfb64baacc4003a110f83ee9208268; no voters:
I20260924 15:27:52.257000 8618 leader_election.cc:288] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:52.257068 8621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:52.257236 8621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [term 1 LEADER]: Becoming Leader. State: Replica: 2cbfb64baacc4003a110f83ee9208268, State: Running, Role: LEADER
I20260924 15:27:52.257275 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:52.257328 8621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } }
I20260924 15:27:52.257714 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cbfb64baacc4003a110f83ee9208268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } } }
I20260924 15:27:52.257793 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:52.257738 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbfb64baacc4003a110f83ee9208268. Latest consensus state: current_term: 1 leader_uuid: "2cbfb64baacc4003a110f83ee9208268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbfb64baacc4003a110f83ee9208268" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43081 } } }
I20260924 15:27:52.257982 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:52.258004 8629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:52.258656 8629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:52.258764 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8550
I20260924 15:27:52.258859 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:52.259063 5515 external_mini_cluster.cc:1428] Setting key d4b49657e0ae1e11c4f5c9ea6eeb7932
I20260924 15:27:52.260296 8629 catalog_manager.cc:1347] Generated new cluster ID: 3effb42ca93f4bbaaa37c8856507e9be
I20260924 15:27:52.260347 8629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:52.270890 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.259540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.5.98.254:43081)
0924 15:27:52.259750 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:52.259757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:52.259985 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:27:52.260137 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:52.260145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:52.260211 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:52.261139 (+ 928us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:52.261244 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.262697 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.263763 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.264394 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.264508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.267285 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:52.267289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:52.267289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:52.267301 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:52.269273 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.269687 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.269690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.269690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.269788 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.270002 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.270003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.270003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.270280 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:52.270424 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:52.270546 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 15:27:52.270672 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260924 15:27:52.275058 8629 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:52.275576 8629 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:52.279690 8629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cbfb64baacc4003a110f83ee9208268: Generated new TSK 0
I20260924 15:27:52.279852 8629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:52.308359 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:52.388221 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:52.388411 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:52.388433 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:52.389500 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:52.389533 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:52.389557 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:52.390894 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:52.391067 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:52.391232 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:52.396457 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.396472 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.396704 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:52.396674 8644 server_base.cc:1034] running on GCE node
I20260924 15:27:52.396900 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:52.397099 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:52.398236 8644 hybrid_clock.cc:648] HybridClock initialized: now 1790263672398216 us; error 39 us; skew 500 ppm
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:52.400875 8644 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:52.401340 8644 webserver.cc:460] Webserver started at http://127.5.98.193:38201/ using document root <none> and password file <none>
I20260924 15:27:52.401574 8644 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:52.401620 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:52.401773 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:52.402823 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "39e254dc90c44293b942c24c2fe75b06"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fd4e0f5ff4ed14f85786790779f85330"
server_key_iv: "5c694c92594dbe0e218878133ba984ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.403183 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "39e254dc90c44293b942c24c2fe75b06"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fd4e0f5ff4ed14f85786790779f85330"
server_key_iv: "5c694c92594dbe0e218878133ba984ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.404580 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:52.405346 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.405566 8644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:52.405671 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "39e254dc90c44293b942c24c2fe75b06"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "fd4e0f5ff4ed14f85786790779f85330"
server_key_iv: "5c694c92594dbe0e218878133ba984ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.405746 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:52.421712 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:52.423842 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:52.423993 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:52.424217 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:52.424515 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:52.424576 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.424619 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:52.424645 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.430343 8644 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:43329
I20260924 15:27:52.430387 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:43329 every 8 connection(s)
I20260924 15:27:52.430688 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:52.432874 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8644
I20260924 15:27:52.432993 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:52.433180 5515 external_mini_cluster.cc:1428] Setting key d7642575dec73ed27dac532d53d2791a
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:52.441134 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.431366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:55993 (local address 127.5.98.254:43081)
0924 15:27:52.431426 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:52.431429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:52.431639 (+ 210us) server_negotiation.cc:407] Connection header received
0924 15:27:52.432156 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:52.432157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:52.432167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:52.432223 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:52.433456 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.434078 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.435575 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.435924 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.436002 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.438261 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:52.438262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:52.438262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:52.438268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:52.439638 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.440176 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.440177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.440177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.440228 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.440508 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.440509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.440509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.440610 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:52.440657 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:52.440991 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 15:27:52.441071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:27:52.441429 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.431317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43081 (local address 127.5.98.193:55993)
0924 15:27:52.431511 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:52.431552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:52.432068 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:52.432321 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:52.432325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:52.433074 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:52.433353 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.433359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.434170 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.434171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.435495 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.435498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.436011 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.436011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.436112 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.436627 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:52.436636 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:52.438165 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:52.439764 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.439771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.439773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.440098 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.440294 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.440295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.440295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.440436 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.440675 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:52.440682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:52.440843 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:27:52.440972 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 15:27:52.441157 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
I20260924 15:27:52.442040 8773 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43081
I20260924 15:27:52.442148 8773 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:52.442370 8773 heartbeater.cc:511] Master 127.5.98.254:43081 requested a full tablet report, sending...
I20260924 15:27:52.442848 8582 ts_manager.cc:194] Registered new tserver with Master: 39e254dc90c44293b942c24c2fe75b06 (127.5.98.193:43329)
I20260924 15:27:52.443473 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:55993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:52.459102 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:52.548324 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:52.548530 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:52.548558 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:52.549715 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:52.549748 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:52.549773 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:52.551090 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:52.551260 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:52.551430 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:52.556352 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.556344 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.556344 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:52.556524 8778 server_base.cc:1034] running on GCE node
I20260924 15:27:52.556759 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:52.557039 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:52.558197 8778 hybrid_clock.cc:648] HybridClock initialized: now 1790263672558184 us; error 39 us; skew 500 ppm
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:52.560665 8778 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:52.561138 8778 webserver.cc:460] Webserver started at http://127.5.98.194:44697/ using document root <none> and password file <none>
I20260924 15:27:52.561312 8778 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:52.561350 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:52.561440 8778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:52.562427 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "5c0564a2701c4974a412ef2403652cc8"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "aea22cfdbc3d2fece0a9979a6d6021ac"
server_key_iv: "8f50c90a3ffbe14c337f6418026350dc"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.562727 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "5c0564a2701c4974a412ef2403652cc8"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "aea22cfdbc3d2fece0a9979a6d6021ac"
server_key_iv: "8f50c90a3ffbe14c337f6418026350dc"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.564010 8778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:52.564718 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.564915 8778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:52.565008 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "5c0564a2701c4974a412ef2403652cc8"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "aea22cfdbc3d2fece0a9979a6d6021ac"
server_key_iv: "8f50c90a3ffbe14c337f6418026350dc"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.565076 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:52.574405 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:52.576798 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:52.576970 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:52.577207 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:52.577536 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:52.577572 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.577615 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:52.577647 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.583830 8778 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40043
I20260924 15:27:52.583900 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40043 every 8 connection(s)
I20260924 15:27:52.584213 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:52.593518 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.584948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:32851 (local address 127.5.98.254:43081)
0924 15:27:52.585016 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:52.585019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:52.585254 (+ 235us) server_negotiation.cc:407] Connection header received
0924 15:27:52.585803 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:52.585804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:52.585816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:52.585875 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:52.586766 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.587266 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.588429 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.588628 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.588697 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.590894 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:52.590896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:52.590896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:52.590903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:52.592288 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.592719 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.592720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.592720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.592774 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.593001 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.593001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.593002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.593104 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:52.593154 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:52.593382 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:27:52.593445 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:27:52.593741 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.584931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43081 (local address 127.5.98.194:32851)
0924 15:27:52.585135 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:52.585171 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:52.585689 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:52.585934 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:52.585937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:52.586492 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:52.586681 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.586686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.587369 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.587370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.588295 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.588298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.588729 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.588730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.588866 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.589403 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:52.589411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:52.590795 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:52.592377 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.592383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.592385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.592634 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.592837 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.592838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.592838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.592934 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.593159 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:52.593163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:52.593273 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 15:27:52.593352 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:27:52.593485 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260924 15:27:52.593955 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8778
I20260924 15:27:52.594096 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:52.594311 5515 external_mini_cluster.cc:1428] Setting key 848806d7961705c6ca83bdb0474a0b86
I20260924 15:27:52.594384 8907 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43081
I20260924 15:27:52.594484 8907 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:52.594673 8907 heartbeater.cc:511] Master 127.5.98.254:43081 requested a full tablet report, sending...
I20260924 15:27:52.595094 8582 ts_manager.cc:194] Registered new tserver with Master: 5c0564a2701c4974a412ef2403652cc8 (127.5.98.194:40043)
I20260924 15:27:52.595614 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:32851
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:52.619307 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:52.700407 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:52.700608 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:52.700631 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:52.701702 8912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:52.701735 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:52.701761 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:52.703115 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:52.703300 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:52.703482 8912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:52.708415 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.708416 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:52.708484 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:52.708835 8912 server_base.cc:1034] running on GCE node
I20260924 15:27:52.709013 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:52.709244 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:52.710382 8912 hybrid_clock.cc:648] HybridClock initialized: now 1790263672710377 us; error 36 us; skew 500 ppm
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:52.712879 8912 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:52.713335 8912 webserver.cc:460] Webserver started at http://127.5.98.195:39517/ using document root <none> and password file <none>
I20260924 15:27:52.713572 8912 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:52.713640 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:52.713788 8912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:52.714802 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "f8c348f5275a46ab87fc657a98889e28"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "82280a5a5e2a45416d171e7354ee56d0"
server_key_iv: "94e5914414382778e0746b687e6e12b7"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.715176 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "f8c348f5275a46ab87fc657a98889e28"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "82280a5a5e2a45416d171e7354ee56d0"
server_key_iv: "94e5914414382778e0746b687e6e12b7"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.716614 8912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:52.717360 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.717571 8912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:52.717675 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "f8c348f5275a46ab87fc657a98889e28"
format_stamp: "Formatted at 2026-09-24 15:27:52 on dist-test-slave-4g6f"
server_key: "82280a5a5e2a45416d171e7354ee56d0"
server_key_iv: "94e5914414382778e0746b687e6e12b7"
server_key_version: "encryptionkey@0"
I20260924 15:27:52.717746 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:52.737035 8912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:52.738991 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:52.739161 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:52.739393 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:52.739712 8912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:52.739765 8912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.739811 8912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:52.739836 8912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:52.745703 8912 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:36353
I20260924 15:27:52.745752 9040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:36353 every 8 connection(s)
I20260924 15:27:52.746109 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:52.754274 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 8912
I20260924 15:27:52.754391 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:52.754645 5515 external_mini_cluster.cc:1428] Setting key a802207074006f6b473d34597ec47cfa
I20260924 15:27:52.755515 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.746875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:50079 (local address 127.5.98.254:43081)
0924 15:27:52.746949 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:52.746951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:52.747132 (+ 181us) server_negotiation.cc:407] Connection header received
0924 15:27:52.747619 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:52.747620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:52.747632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:52.747756 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:52.748676 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.749205 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.750183 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.750377 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.750443 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.752585 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:52.752587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:52.752587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:52.752599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:52.753908 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.754401 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.754402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.754403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.754470 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.754774 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.754774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.754775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.754925 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:52.755010 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:52.755340 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 15:27:52.755420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:27:52.755766 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.746819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43081 (local address 127.5.98.195:50079)
0924 15:27:52.746994 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:52.747036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:52.747521 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:52.747809 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:52.747812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:52.748366 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:52.748558 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.748564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.749290 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.749290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.750073 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:52.750075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.750458 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.750459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:52.750568 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.751002 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:52.751010 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:52.752499 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:52.753994 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.754000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.754002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.754323 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.754559 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:52.754560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:52.754561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:52.754708 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:52.755002 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:52.755009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:52.755193 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 15:27:52.755309 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 15:27:52.755451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260924 15:27:52.756230 9041 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43081
I20260924 15:27:52.756340 9041 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:52.756531 9041 heartbeater.cc:511] Master 127.5.98.254:43081 requested a full tablet report, sending...
I20260924 15:27:52.756901 8582 ts_manager.cc:194] Registered new tserver with Master: f8c348f5275a46ab87fc657a98889e28 (127.5.98.195:36353)
I20260924 15:27:52.757263 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:50079
I20260924 15:27:52.766613 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263672, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:52.788683 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:52.781850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.5.98.254:43081)
0924 15:27:52.781933 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:52.781936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:52.782024 (+ 88us) server_negotiation.cc:407] Connection header received
0924 15:27:52.782112 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:52.782113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:52.782123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:52.782195 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:52.782800 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.783292 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.784127 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:52.784320 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:52.784424 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:52.786508 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:52.786509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:52.786510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:52.786516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:52.787715 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.788033 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.788033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.788034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.788082 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:52.788238 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:52.788239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:52.788239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:52.788376 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:52.788449 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:52.788549 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:27:52.788619 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 15:27:52.788923 8582 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60068
I20260924 15:27:52.789397 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8644
I20260924 15:27:52.795625 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8778
I20260924 15:27:52.800071 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8912
I20260924 15:27:52.804684 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 8550
2026-09-24T15:27:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1851 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[9053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[9053](info): set up 2 sockets
Sep 24 15:27:52 dist-test-slave-4g6f krb5kdc[9053](info): commencing operation
krb5kdc: starting...
W20260924 15:27:53.844643 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:53 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:53.962342 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41395
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:54.042107 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:54.042327 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:54.042364 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:54.043505 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:54.043568 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:54.043591 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:54.043612 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:54.043644 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:54.045166 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:54.045440 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:54.045740 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:54.050911 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.050949 9066 server_base.cc:1034] running on GCE node
W20260924 15:27:54.051012 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.051162 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.051407 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:54.051739 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:54.052906 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790263674052885 us; error 32 us; skew 500 ppm
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:54.055357 9066 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:54.055904 9066 webserver.cc:460] Webserver started at http://127.5.98.254:40919/ using document root <none> and password file <none>
I20260924 15:27:54.056151 9066 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:54.056221 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:54.056372 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:54.057399 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "709a1d51aa1141d18ed94c9ac727da3e"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "f8fc6647bf679ce806b08033a902472e"
server_key_iv: "625962537a30f974d0bd4fe9ad9efd50"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.057763 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "709a1d51aa1141d18ed94c9ac727da3e"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "f8fc6647bf679ce806b08033a902472e"
server_key_iv: "625962537a30f974d0bd4fe9ad9efd50"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.059087 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:54.059859 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.060062 9066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:54.060163 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "709a1d51aa1141d18ed94c9ac727da3e"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "f8fc6647bf679ce806b08033a902472e"
server_key_iv: "625962537a30f974d0bd4fe9ad9efd50"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.060218 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:54.075745 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:54.077603 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:54.077718 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:54.081509 9066 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:41395
I20260924 15:27:54.081585 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:41395 every 8 connection(s)
I20260924 15:27:54.082039 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:54.082687 9134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:54.084909 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e: Bootstrap starting.
I20260924 15:27:54.085493 9134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:54.085780 9134 log.cc:824] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:54.086438 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e: No bootstrap required, opened a new log
I20260924 15:27:54.086772 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9066
I20260924 15:27:54.086884 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:54.087061 5515 external_mini_cluster.cc:1428] Setting key d2d64c6d954db6c22c9aaa1983286d04
I20260924 15:27:54.089058 9134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } }
I20260924 15:27:54.089236 9134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:54.089277 9134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709a1d51aa1141d18ed94c9ac727da3e, State: Initialized, Role: FOLLOWER
I20260924 15:27:54.089417 9134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } }
I20260924 15:27:54.089509 9134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:54.089546 9134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:54.089597 9134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:54.090231 9134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } }
I20260924 15:27:54.090327 9134 leader_election.cc:302] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709a1d51aa1141d18ed94c9ac727da3e; no voters:
I20260924 15:27:54.090535 9134 leader_election.cc:288] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:54.090566 9139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:54.090668 9139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [term 1 LEADER]: Becoming Leader. State: Replica: 709a1d51aa1141d18ed94c9ac727da3e, State: Running, Role: LEADER
I20260924 15:27:54.090757 9139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } }
I20260924 15:27:54.090799 9134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:54.091069 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709a1d51aa1141d18ed94c9ac727da3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } } }
I20260924 15:27:54.091140 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:54.091259 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709a1d51aa1141d18ed94c9ac727da3e. Latest consensus state: current_term: 1 leader_uuid: "709a1d51aa1141d18ed94c9ac727da3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709a1d51aa1141d18ed94c9ac727da3e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41395 } } }
I20260924 15:27:54.091326 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:54.091610 9144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:54.092140 9144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:54.093677 9144 catalog_manager.cc:1347] Generated new cluster ID: f2a7d25238ad4e2d8f34ae9de3d746c5
I20260924 15:27:54.093727 9144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:54.098877 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.087540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.5.98.254:41395)
0924 15:27:54.087740 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.087748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.087772 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:27:54.088991 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.088998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.089047 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.089169 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.090000 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.091285 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.092579 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.093148 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.093297 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.095572 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.095576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.095576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.095587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.097353 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.097729 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.097732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.097733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.097823 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.098038 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.098039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.098039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.098274 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.098435 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.098531 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.098681 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260924 15:27:54.117282 9144 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:54.117837 9144 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:54.124148 9144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709a1d51aa1141d18ed94c9ac727da3e: Generated new TSK 0
I20260924 15:27:54.124338 9144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:54.158634 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:54.239212 9160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:54.239405 9160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:54.239427 9160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:54.240576 9160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:54.240617 9160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:54.240643 9160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:54.242053 9160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:54.242242 9160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:54.242412 9160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:54.247448 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.247449 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.247459 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.247720 9160 server_base.cc:1034] running on GCE node
I20260924 15:27:54.247977 9160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:54.248222 9160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:54.249366 9160 hybrid_clock.cc:648] HybridClock initialized: now 1790263674249352 us; error 33 us; skew 500 ppm
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:54.251924 9160 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:54.252297 9160 webserver.cc:460] Webserver started at http://127.5.98.193:41899/ using document root <none> and password file <none>
I20260924 15:27:54.252494 9160 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:54.252532 9160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:54.252696 9160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:54.253732 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "b0fbe62129594665867a33c38789a703"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "4c13e690bb435fe77a303d800d476ad5"
server_key_iv: "62ad4bb434cacd80bbf101c8e810dad9"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.254033 9160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "b0fbe62129594665867a33c38789a703"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "4c13e690bb435fe77a303d800d476ad5"
server_key_iv: "62ad4bb434cacd80bbf101c8e810dad9"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.255291 9160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:54.256045 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.256229 9160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.256322 9160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "b0fbe62129594665867a33c38789a703"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "4c13e690bb435fe77a303d800d476ad5"
server_key_iv: "62ad4bb434cacd80bbf101c8e810dad9"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.256407 9160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:54.265187 9160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:54.267403 9160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:54.267562 9160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:54.267809 9160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:54.268112 9160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:54.268172 9160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.268211 9160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:54.268239 9160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.274034 9160 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:44881
I20260924 15:27:54.274075 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:44881 every 8 connection(s)
I20260924 15:27:54.274379 9160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:54.283160 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9160
I20260924 15:27:54.283262 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.275055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:33319 (local address 127.5.98.254:41395)
0924 15:27:54.275131 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.275133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.275345 (+ 212us) server_negotiation.cc:407] Connection header received
0924 15:27:54.276331 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.276332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.276343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.276401 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.276966 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.277473 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.278450 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.278645 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.278697 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.280713 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.280715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.280715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.280723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.281884 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.282279 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.282281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.282281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.282343 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.282597 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.282598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.282599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.282713 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.282785 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.283088 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.283161 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:27:54.283280 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:54.283475 5515 external_mini_cluster.cc:1428] Setting key 6639ccba916975cd501a17aa276d40ff
I20260924 15:27:54.283516 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.275040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41395 (local address 127.5.98.193:33319)
0924 15:27:54.275216 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:54.275258 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:54.276239 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:54.276461 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:54.276464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:54.276701 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:54.276892 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.276897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.277553 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.277554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.278343 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.278354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.278717 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.278717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.278813 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.279221 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:54.279229 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:54.280607 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:54.281971 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.281978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.281980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.282199 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.282418 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.282419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.282419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.282522 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.282788 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:54.282795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:54.282925 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 15:27:54.283044 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:27:54.283213 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260924 15:27:54.283996 9289 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41395
I20260924 15:27:54.284109 9289 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:54.284300 9289 heartbeater.cc:511] Master 127.5.98.254:41395 requested a full tablet report, sending...
I20260924 15:27:54.284843 9097 ts_manager.cc:194] Registered new tserver with Master: b0fbe62129594665867a33c38789a703 (127.5.98.193:44881)
I20260924 15:27:54.285547 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:33319
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:54.306304 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:54.386108 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:54.386299 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:54.386322 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:54.387480 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:54.387521 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:54.387576 9294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:54.389082 9294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:54.389320 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:54.389552 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:54.394806 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.395067 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.395078 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.395138 9294 server_base.cc:1034] running on GCE node
I20260924 15:27:54.395350 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:54.395587 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:54.396754 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790263674396743 us; error 31 us; skew 500 ppm
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:54.399133 9294 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:54.399499 9294 webserver.cc:460] Webserver started at http://127.5.98.194:44673/ using document root <none> and password file <none>
I20260924 15:27:54.399789 9294 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:54.399838 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:54.399974 9294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:54.401062 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "da97f30e521d475bb221f68097e733f1"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "62631cb9b19b0963ddb82c927ce9032b"
server_key_iv: "1ca7bf85dc91cd7c15c74fb71cc4a4f4"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.401428 9294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "da97f30e521d475bb221f68097e733f1"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "62631cb9b19b0963ddb82c927ce9032b"
server_key_iv: "1ca7bf85dc91cd7c15c74fb71cc4a4f4"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.402753 9294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:54.403541 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.403816 9294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:54.403889 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "da97f30e521d475bb221f68097e733f1"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "62631cb9b19b0963ddb82c927ce9032b"
server_key_iv: "1ca7bf85dc91cd7c15c74fb71cc4a4f4"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.403986 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:54.417415 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:54.419343 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:54.419483 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:54.419821 9294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:54.420140 9294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:54.420176 9294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.420197 9294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:54.420210 9294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.426069 9294 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46831
I20260924 15:27:54.426141 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46831 every 8 connection(s)
I20260924 15:27:54.426455 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:54.430724 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9294
I20260924 15:27:54.430841 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:54.431073 5515 external_mini_cluster.cc:1428] Setting key 484936939bb12349f79206b856c32901
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:54.436833 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.427180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:39663 (local address 127.5.98.254:41395)
0924 15:27:54.427225 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.427227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.427470 (+ 243us) server_negotiation.cc:407] Connection header received
0924 15:27:54.428398 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.428400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.428413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.428491 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.429058 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.429625 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.430584 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.430793 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.430950 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.433569 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.433571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.433571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.433577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.435171 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.435691 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.435693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.435693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.435772 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.436065 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.436066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.436067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.436204 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.436311 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.436623 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.436727 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:54.437083 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.427142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41395 (local address 127.5.98.194:39663)
0924 15:27:54.427297 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:54.427337 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:54.428289 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:54.428546 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:54.428549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:54.428788 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:54.428971 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.428975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.429713 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.429714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.430478 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.430480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.430959 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.430960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.431109 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.431675 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:54.431686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:54.433466 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:54.435292 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.435301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.435303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.435565 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.435855 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.435856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.435856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.435981 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.436288 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:54.436307 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:54.436469 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 15:27:54.436596 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 15:27:54.436789 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260924 15:27:54.437662 9423 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41395
I20260924 15:27:54.437785 9423 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:54.438001 9423 heartbeater.cc:511] Master 127.5.98.254:41395 requested a full tablet report, sending...
I20260924 15:27:54.438460 9097 ts_manager.cc:194] Registered new tserver with Master: da97f30e521d475bb221f68097e733f1 (127.5.98.194:46831)
I20260924 15:27:54.438988 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:39663
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:54.456665 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:54.537297 9428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:54.537484 9428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:54.537506 9428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:54.538563 9428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:54.538597 9428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:54.538622 9428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:54.540088 9428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:54.540287 9428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:54.540441 9428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:54.545315 9433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.545401 9428 server_base.cc:1034] running on GCE node
W20260924 15:27:54.545493 9436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:54.545462 9434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:54.545802 9428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:54.546036 9428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:54.547204 9428 hybrid_clock.cc:648] HybridClock initialized: now 1790263674547192 us; error 44 us; skew 500 ppm
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:54.549678 9428 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:54.550050 9428 webserver.cc:460] Webserver started at http://127.5.98.195:35193/ using document root <none> and password file <none>
I20260924 15:27:54.550230 9428 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:54.550269 9428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:54.550354 9428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:54.551287 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "2720f43990d644c9ab205a55c8fd5e31"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "5a0d616cb90ca1fdb38dc135eafeba3b"
server_key_iv: "1467e606259bc19e6f477e1d8b9b704b"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.551578 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "2720f43990d644c9ab205a55c8fd5e31"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "5a0d616cb90ca1fdb38dc135eafeba3b"
server_key_iv: "1467e606259bc19e6f477e1d8b9b704b"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.552870 9428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:54.553529 9443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.553680 9428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:54.553766 9428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "2720f43990d644c9ab205a55c8fd5e31"
format_stamp: "Formatted at 2026-09-24 15:27:54 on dist-test-slave-4g6f"
server_key: "5a0d616cb90ca1fdb38dc135eafeba3b"
server_key_iv: "1467e606259bc19e6f477e1d8b9b704b"
server_key_version: "encryptionkey@0"
I20260924 15:27:54.553819 9428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:54.570348 9428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:54.572284 9428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:54.572394 9428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:54.572559 9428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:54.572885 9428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:54.572916 9428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.572937 9428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:54.572986 9428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:54.578428 9428 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:37551
I20260924 15:27:54.578514 9556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:37551 every 8 connection(s)
I20260924 15:27:54.578827 9428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:54.581295 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9428
I20260924 15:27:54.581384 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:54.581561 5515 external_mini_cluster.cc:1428] Setting key 70274b4693268bd799a7eb1fc0d49011
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:54.588125 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.579563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:37523 (local address 127.5.98.254:41395)
0924 15:27:54.579627 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.579629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.579806 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:27:54.580775 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.580776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.580789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.580841 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.581508 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.582185 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.583244 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.583441 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.583535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.585558 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.585559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.585560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.585566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.586937 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.587281 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.587282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.587282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.587348 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.587564 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.587564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.587565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.587728 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.587781 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.588004 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.588060 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:27:54.588337 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.579506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41395 (local address 127.5.98.195:37523)
0924 15:27:54.579683 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:54.579721 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:54.580659 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:54.580916 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:54.580919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:54.581160 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:54.581359 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.581365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.582259 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.582260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.583148 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:54.583151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.583552 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.583553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:54.583727 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.584145 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:54.584153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:54.585477 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:54.587009 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.587014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.587016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.587217 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.587411 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:54.587411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:54.587412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:54.587509 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:54.587784 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:54.587789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:54.587895 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 15:27:54.587977 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:27:54.588100 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260924 15:27:54.588848 9557 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41395
I20260924 15:27:54.588956 9557 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:54.589125 9557 heartbeater.cc:511] Master 127.5.98.254:41395 requested a full tablet report, sending...
I20260924 15:27:54.589462 9097 ts_manager.cc:194] Registered new tserver with Master: 2720f43990d644c9ab205a55c8fd5e31 (127.5.98.195:37551)
I20260924 15:27:54.589825 9097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:37523
I20260924 15:27:54.593319 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:27:54.615404 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.607956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.5.98.193:44881)
0924 15:27:54.608121 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.608126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.608145 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:27:54.608171 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.608173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.608186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.608252 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.608782 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.609508 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.610206 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.610524 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.610585 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.612829 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.612837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.612837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.612844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.614380 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.614679 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.614681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.614682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.614734 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.614952 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.614953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.614953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.615084 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.615206 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.615265 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.615326 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:54.623214 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:54.616737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.5.98.254:41395)
0924 15:27:54.616797 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:54.616799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:54.616807 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:54.616873 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:54.616874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:54.616885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:54.616932 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:54.617381 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.617872 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.618562 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:54.618752 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:54.618815 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:54.621029 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:54.621030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:54.621031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:54.621038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:54.622222 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.622524 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.622525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.622525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.622577 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:54.622771 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:54.622772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:54.622773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:54.622919 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:54.623007 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:54.623091 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:27:54.623151 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 15:27:54.623520 9097 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39494
I20260924 15:27:54.623950 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9160
I20260924 15:27:54.629927 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9294
I20260924 15:27:54.634507 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9428
I20260924 15:27:54.638684 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9066
2026-09-24T15:27:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1833 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9569](info): set up 2 sockets
Sep 24 15:27:54 dist-test-slave-4g6f krb5kdc[9569](info): commencing operation
krb5kdc: starting...
W20260924 15:27:55.672713 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:55 dist-test-slave-4g6f krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:55.794063 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:39303
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:55.872752 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:55.873011 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:55.873047 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:55.874122 9582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:55.874187 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:55.874210 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:55.874235 9582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:55.874265 9582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:55.875751 9582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:39303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:55.876021 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:55.876240 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:55.880859 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:55.880858 9587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:55.880882 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:55.881112 9582 server_base.cc:1034] running on GCE node
I20260924 15:27:55.881343 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:55.881577 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:55.882716 9582 hybrid_clock.cc:648] HybridClock initialized: now 1790263675882703 us; error 28 us; skew 500 ppm
Sep 24 15:27:55 dist-test-slave-4g6f krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:55.885062 9582 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:55.885396 9582 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:55.885465 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:55.885624 9582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:55.886669 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "3f16b747b8824165b1512c83383d48fb"
format_stamp: "Formatted at 2026-09-24 15:27:55 on dist-test-slave-4g6f"
server_key: "39428b929bcd09634dcfed5652959dc3"
server_key_iv: "5ac95f421f356ed7df5783bbc8857193"
server_key_version: "encryptionkey@0"
I20260924 15:27:55.887024 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "3f16b747b8824165b1512c83383d48fb"
format_stamp: "Formatted at 2026-09-24 15:27:55 on dist-test-slave-4g6f"
server_key: "39428b929bcd09634dcfed5652959dc3"
server_key_iv: "5ac95f421f356ed7df5783bbc8857193"
server_key_version: "encryptionkey@0"
I20260924 15:27:55.888366 9582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:55.889062 9596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:55.889236 9582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:55.889334 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "3f16b747b8824165b1512c83383d48fb"
format_stamp: "Formatted at 2026-09-24 15:27:55 on dist-test-slave-4g6f"
server_key: "39428b929bcd09634dcfed5652959dc3"
server_key_iv: "5ac95f421f356ed7df5783bbc8857193"
server_key_version: "encryptionkey@0"
I20260924 15:27:55.889418 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:55.902004 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:55.903836 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:55.903995 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:55.907758 9582 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:39303
I20260924 15:27:55.907812 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:39303 every 8 connection(s)
I20260924 15:27:55.908108 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:55.908663 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9582
I20260924 15:27:55.908780 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:55.908962 5515 external_mini_cluster.cc:1428] Setting key 1368a1b8b1e7234967e5c77c78bfb7e9
I20260924 15:27:55.909242 9649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:55.912120 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: Bootstrap starting.
I20260924 15:27:55.912737 9649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:55.913020 9649 log.cc:824] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:55.913692 9649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: No bootstrap required, opened a new log
Sep 24 15:27:55 dist-test-slave-4g6f krb5kdc[9569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:55.915879 9649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } }
I20260924 15:27:55.916010 9649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:55.916033 9649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f16b747b8824165b1512c83383d48fb, State: Initialized, Role: FOLLOWER
I20260924 15:27:55.916170 9649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } }
I20260924 15:27:55.916267 9649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:55.916311 9649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:55.916381 9649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:55.917099 9649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } }
I20260924 15:27:55.917212 9649 leader_election.cc:302] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f16b747b8824165b1512c83383d48fb; no voters:
I20260924 15:27:55.917419 9649 leader_election.cc:288] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:55.917446 9654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:55.917555 9654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [term 1 LEADER]: Becoming Leader. State: Replica: 3f16b747b8824165b1512c83383d48fb, State: Running, Role: LEADER
I20260924 15:27:55.917645 9654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } }
I20260924 15:27:55.917691 9649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:55.917974 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f16b747b8824165b1512c83383d48fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } } }
I20260924 15:27:55.918128 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:55.918318 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f16b747b8824165b1512c83383d48fb. Latest consensus state: current_term: 1 leader_uuid: "3f16b747b8824165b1512c83383d48fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f16b747b8824165b1512c83383d48fb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39303 } } }
I20260924 15:27:55.918396 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:55.918752 9661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:55.919359 9661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:55.920199 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:55.909443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.5.98.254:39303)
0924 15:27:55.909641 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:55.909651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:55.909682 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:27:55.910372 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:55.910380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:55.910430 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:55.910583 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:55.911232 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:55.912251 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:55.913100 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:55.913538 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:55.913660 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:55.916303 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:55.916307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:55.916308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:55.916321 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:55.918315 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:55.918757 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:55.918760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:55.918760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:55.918868 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:55.919108 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:55.919109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:55.919109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:55.919383 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:55.919586 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:55.919708 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 15:27:55.919938 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260924 15:27:55.921031 9661 catalog_manager.cc:1347] Generated new cluster ID: 3642d6ebed7146f98266bd9be45dbb0f
I20260924 15:27:55.921113 9661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:55.931361 9661 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:55.931941 9661 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:55.936542 9661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: Generated new TSK 0
I20260924 15:27:55.936723 9661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:55.965986 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:56.066776 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:56.066979 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:56.067003 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:56.068184 9675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:56.068224 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:56.068251 9675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:56.069748 9675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:56.069990 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:56.070245 9675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:56.076160 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.076189 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.076166 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:56.076530 9675 server_base.cc:1034] running on GCE node
I20260924 15:27:56.076746 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:56.076989 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:56.078151 9675 hybrid_clock.cc:648] HybridClock initialized: now 1790263676078127 us; error 34 us; skew 500 ppm
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:56.080725 9675 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:56.081053 9675 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:56.081130 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:56.081279 9675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:56.082314 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "da1e8b8773944471adaba761bf8bfe47"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "133aebea3878a1a8c9231726f6d56ef1"
server_key_iv: "08074a8bec8937a130043efca524d2a7"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.082693 9675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "da1e8b8773944471adaba761bf8bfe47"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "133aebea3878a1a8c9231726f6d56ef1"
server_key_iv: "08074a8bec8937a130043efca524d2a7"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.084016 9675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:56.084725 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.084885 9675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:56.084990 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "da1e8b8773944471adaba761bf8bfe47"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "133aebea3878a1a8c9231726f6d56ef1"
server_key_iv: "08074a8bec8937a130043efca524d2a7"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.085062 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:56.139379 9675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:56.141372 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:56.141547 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:56.141824 9675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:56.142165 9675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:56.142222 9675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.142261 9675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:56.142289 9675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.148073 9675 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:36043
I20260924 15:27:56.148121 9802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:36043 every 8 connection(s)
I20260924 15:27:56.148383 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:56.151813 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9675
I20260924 15:27:56.151939 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:56.152117 5515 external_mini_cluster.cc:1428] Setting key 3910c1c012528b82e3093d0cdcff44db
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:56.158290 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.149078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:39303 (local address 127.5.98.193:37865)
0924 15:27:56.149249 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.149284 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.149782 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.150054 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.150058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.150296 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:56.150775 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.150780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.151510 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.151513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.152463 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.152466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.152879 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.152880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.153004 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.153490 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:56.153498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:56.155067 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:56.156927 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.156933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.156935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.157152 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.157365 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.157365 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.157366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.157463 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.157759 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:56.157763 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:56.157912 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.158012 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.158133 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260924 15:27:56.158299 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.149089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:37865 (local address 127.5.98.254:39303)
0924 15:27:56.149162 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.149164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.149371 (+ 207us) server_negotiation.cc:407] Connection header received
0924 15:27:56.149890 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.149892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.149904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.149959 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:56.150881 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.151388 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.152554 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.152782 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.152867 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.155186 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:56.155187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:56.155187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:56.155193 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:56.156841 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.157230 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.157232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.157232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.157295 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.157536 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.157537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.157538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.157687 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:56.157767 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.158059 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.158162 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:27:56.158833 9803 heartbeater.cc:347] Connected to a master server at 127.5.98.254:39303
I20260924 15:27:56.159046 9803 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:56.159219 9803 heartbeater.cc:511] Master 127.5.98.254:39303 requested a full tablet report, sending...
I20260924 15:27:56.159830 9613 ts_manager.cc:194] Registered new tserver with Master: da1e8b8773944471adaba761bf8bfe47 (127.5.98.193:36043)
I20260924 15:27:56.160778 9613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:37865
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:56.179080 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:56.259250 9808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:56.259442 9808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:56.259466 9808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:56.260668 9808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:56.260711 9808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:56.260738 9808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:56.262224 9808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:56.262458 9808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:56.262715 9808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:56.267966 9814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.267995 9816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:56.268033 9808 server_base.cc:1034] running on GCE node
W20260924 15:27:56.267995 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:56.268359 9808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:56.268653 9808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:56.269840 9808 hybrid_clock.cc:648] HybridClock initialized: now 1790263676269824 us; error 46 us; skew 500 ppm
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:56.272698 9808 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:27:56.273128 9808 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:56.273208 9808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:56.273393 9808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:56.274880 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "516e3e8895394a2e82841cbf0d15fde2"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "f344ffe0027d6f59e8bbd7c66c67f046"
server_key_iv: "c0c06be473bfb2d62f153aae7e012ff3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.275305 9808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "516e3e8895394a2e82841cbf0d15fde2"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "f344ffe0027d6f59e8bbd7c66c67f046"
server_key_iv: "c0c06be473bfb2d62f153aae7e012ff3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.276779 9808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:56.277552 9822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.277762 9808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:56.277868 9808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "516e3e8895394a2e82841cbf0d15fde2"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "f344ffe0027d6f59e8bbd7c66c67f046"
server_key_iv: "c0c06be473bfb2d62f153aae7e012ff3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.277959 9808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:56.292500 9808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:56.294608 9808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:56.294780 9808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:56.295019 9808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:56.295352 9808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:56.295419 9808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.295465 9808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:56.295496 9808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.301813 9808 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:32993
I20260924 15:27:56.301847 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:32993 every 8 connection(s)
I20260924 15:27:56.302129 9808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:56.303499 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9808
I20260924 15:27:56.303627 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:56.303848 5515 external_mini_cluster.cc:1428] Setting key d96ed5ca28574573c291fdec464dda6c
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:56.312466 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.302814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:39303 (local address 127.5.98.194:54353)
0924 15:27:56.302976 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.303010 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.303441 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.303715 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.303719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.304023 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:56.304614 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.304620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.305293 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.305294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.306415 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.306419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.306819 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.306820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.306989 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.307510 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:56.307519 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:56.309213 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:56.311116 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.311121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.311123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.311325 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.311555 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.311556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.311556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.311694 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.312016 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:56.312021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:56.312129 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.312204 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.312315 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260924 15:27:56.312512 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.302860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:54353 (local address 127.5.98.254:39303)
0924 15:27:56.302941 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.302944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.303094 (+ 150us) server_negotiation.cc:407] Connection header received
0924 15:27:56.303545 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.303546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.303560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.303610 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:56.304710 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.305199 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.306509 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.306721 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.306787 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.309316 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:56.309318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:56.309318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:56.309328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:56.311027 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.311405 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.311407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.311407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.311478 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.311771 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.311772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.311773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.311941 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:56.312029 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.312281 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.312373 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:27:56.313009 9936 heartbeater.cc:347] Connected to a master server at 127.5.98.254:39303
I20260924 15:27:56.313118 9936 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:56.313279 9936 heartbeater.cc:511] Master 127.5.98.254:39303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:27:56.313654 9613 ts_manager.cc:194] Registered new tserver with Master: 516e3e8895394a2e82841cbf0d15fde2 (127.5.98.194:32993)
I20260924 15:27:56.314178 9613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:54353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:27:56.332063 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:56.414506 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:56.414716 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:56.414741 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:56.416306 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:56.416352 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:56.416388 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:56.417850 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:56.418123 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:56.418296 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:56.423269 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.423264 9949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.423264 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:56.423516 9941 server_base.cc:1034] running on GCE node
I20260924 15:27:56.423732 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:56.423938 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:56.425100 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790263676425090 us; error 40 us; skew 500 ppm
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:56.427402 9941 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:27:56.427817 9941 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:56.427894 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:56.428035 9941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:56.429154 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "94f8996d4cbe488c801f1031532de8c9"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "3f80f8d39d7f521894acf3dbcdd5b3cd"
server_key_iv: "af45a2dfd03e10f14ab14725f7d91fe3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.429576 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "94f8996d4cbe488c801f1031532de8c9"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "3f80f8d39d7f521894acf3dbcdd5b3cd"
server_key_iv: "af45a2dfd03e10f14ab14725f7d91fe3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.430842 9941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:56.431519 9955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.431742 9941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:56.431840 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "94f8996d4cbe488c801f1031532de8c9"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "3f80f8d39d7f521894acf3dbcdd5b3cd"
server_key_iv: "af45a2dfd03e10f14ab14725f7d91fe3"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.431921 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:56.449596 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:56.451305 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:56.451469 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:56.451748 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:56.452062 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:56.452116 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.452158 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:56.452184 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.457890 9941 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42793
I20260924 15:27:56.457940 10069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42793 every 8 connection(s)
I20260924 15:27:56.458194 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:27:56 dist-test-slave-4g6f krb5kdc[9569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:56.466867 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 9941
I20260924 15:27:56.466974 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:56.467017 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.458890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:33601 (local address 127.5.98.254:39303)
0924 15:27:56.458936 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.458938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.459149 (+ 211us) server_negotiation.cc:407] Connection header received
0924 15:27:56.459697 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.459699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.459712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.459770 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:56.460700 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.461216 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.462213 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.462430 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.462516 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.464644 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:56.464646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:56.464646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:56.464653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:56.465807 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.466167 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.466168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.466168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.466218 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.466421 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.466421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.466421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.466518 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:56.466568 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.466814 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.466902 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:27:56.467146 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.458857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:39303 (local address 127.5.98.195:33601)
0924 15:27:56.459030 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.459065 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.459556 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.459844 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.459848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.460107 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:56.460587 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.460600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.461314 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.461315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.462119 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.462121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.462525 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.462526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.462658 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.463165 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:56.463174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:56.464559 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:56.465877 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.465883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.465885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.466104 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.466275 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:56.466276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:56.466276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:56.466369 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:56.466571 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:56.466575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:56.466682 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.466764 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.466875 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260924 15:27:56.467206 5515 external_mini_cluster.cc:1428] Setting key 15aad2f9b7557832be86d9f1e7ff99e7
I20260924 15:27:56.467693 10070 heartbeater.cc:347] Connected to a master server at 127.5.98.254:39303
I20260924 15:27:56.467792 10070 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:56.467940 10070 heartbeater.cc:511] Master 127.5.98.254:39303 requested a full tablet report, sending...
I20260924 15:27:56.468297 9612 ts_manager.cc:194] Registered new tserver with Master: 94f8996d4cbe488c801f1031532de8c9 (127.5.98.195:42793)
I20260924 15:27:56.468652 9612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:33601
I20260924 15:27:56.478847 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:56.486456 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.480072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.5.98.254:39303)
0924 15:27:56.480165 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.480172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.480219 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:27:56.480268 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.480269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.480277 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.480323 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:56.480948 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.481518 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.482291 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.482526 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.482595 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.483740 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:56.483742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:56.483742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:56.483757 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:56.485340 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.485671 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.485672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.485672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.485731 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:56.485888 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:56.485889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:56.485889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:56.486030 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:56.486114 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.486203 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.486268 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 15:27:56.488300 9612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:56.489849 9612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:56.496484 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.5.98.194:32993)
0924 15:27:56.492728 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.492734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.492763 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:27:56.492922 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.492925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.492943 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.493176 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:56.493258 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.494303 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.495565 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496117 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496207 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.496354 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.496393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260924 15:27:56.496471 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36043 (local address 127.0.0.1:43274)
0924 15:27:56.492338 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.492349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.492480 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.492806 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.492810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.492812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:56.492881 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.492883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.494264 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.494265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.495215 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.495218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496119 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.496207 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496224 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.496268 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.496324 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260924 15:27:56.496639 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.5.98.193:36043)
0924 15:27:56.492609 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.492615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.492627 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:27:56.492678 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.492680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.492696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.492786 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:56.492985 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.494156 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.495294 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496017 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496431 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496493 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.496516 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.496550 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260924 15:27:56.496634 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.5.98.195:42793)
0924 15:27:56.492655 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.492660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.492869 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:27:56.492913 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.492915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.492931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.493057 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:56.493270 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.493985 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.495421 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496153 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496211 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496268 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.496514 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.496558 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260924 15:27:56.496673 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:32993 (local address 127.0.0.1:45100)
0924 15:27:56.492297 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.492336 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.492402 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.493047 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.493048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.493051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:56.493102 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.493104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.494403 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.494404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.495486 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.495488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496210 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.496275 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496280 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.496339 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.496383 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":43,"threads_started":1}
I20260924 15:27:56.496747 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.492391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42793 (local address 127.0.0.1:54866)
0924 15:27:56.492754 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.492764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.492813 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.493136 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.493137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.493139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:56.493194 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.493195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.494087 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.494088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.495319 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.495322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.496300 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.496301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.496350 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.496354 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.496400 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.496437 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":97,"threads_started":1}
I20260924 15:27:56.497506 9737 tablet_service.cc:1467] Processing CreateTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 (DEFAULT_TABLE table=test-table [id=e070b9d4cde14b06872e09d366211820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:56.497567 9870 tablet_service.cc:1467] Processing CreateTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 (DEFAULT_TABLE table=test-table [id=e070b9d4cde14b06872e09d366211820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:56.497534 10003 tablet_service.cc:1467] Processing CreateTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 (DEFAULT_TABLE table=test-table [id=e070b9d4cde14b06872e09d366211820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:56.497848 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c6b678a7a346e888cb08fb7ed30228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:56.497865 9870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c6b678a7a346e888cb08fb7ed30228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:56.497865 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c6b678a7a346e888cb08fb7ed30228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:56.500339 10087 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Bootstrap starting.
I20260924 15:27:56.500926 10087 tablet_bootstrap.cc:654] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:56.501004 10088 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Bootstrap starting.
I20260924 15:27:56.501248 10089 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Bootstrap starting.
I20260924 15:27:56.501713 10088 tablet_bootstrap.cc:654] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:56.501756 10087 log.cc:824] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:56.501881 10089 tablet_bootstrap.cc:654] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:56.502311 10088 log.cc:824] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:56.502409 10089 log.cc:824] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:56.503111 10087 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: No bootstrap required, opened a new log
I20260924 15:27:56.503122 10088 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: No bootstrap required, opened a new log
I20260924 15:27:56.503188 10088 ts_tablet_manager.cc:1397] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:56.503188 10087 ts_tablet_manager.cc:1397] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:27:56.504051 10089 tablet_bootstrap.cc:492] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: No bootstrap required, opened a new log
I20260924 15:27:56.504133 10089 ts_tablet_manager.cc:1397] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:27:56.505189 10087 raft_consensus.cc:348] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505190 10088 raft_consensus.cc:348] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505360 10087 raft_consensus.cc:374] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:56.505360 10088 raft_consensus.cc:374] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:56.505398 10088 raft_consensus.cc:729] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f8996d4cbe488c801f1031532de8c9, State: Initialized, Role: FOLLOWER
I20260924 15:27:56.505398 10087 raft_consensus.cc:729] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1e8b8773944471adaba761bf8bfe47, State: Initialized, Role: FOLLOWER
I20260924 15:27:56.505528 10087 consensus_queue.cc:260] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505528 10088 consensus_queue.cc:260] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505496 10089 raft_consensus.cc:348] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505685 10089 raft_consensus.cc:374] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:56.505725 10089 raft_consensus.cc:729] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516e3e8895394a2e82841cbf0d15fde2, State: Initialized, Role: FOLLOWER
I20260924 15:27:56.505812 10088 ts_tablet_manager.cc:1428] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:56.505812 10087 ts_tablet_manager.cc:1428] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:56.505841 10089 consensus_queue.cc:260] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.505949 10070 heartbeater.cc:503] Master 127.5.98.254:39303 was elected leader, sending a full tablet report...
I20260924 15:27:56.506115 10089 ts_tablet_manager.cc:1428] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:56.506145 9936 heartbeater.cc:503] Master 127.5.98.254:39303 was elected leader, sending a full tablet report...
I20260924 15:27:56.506217 9803 heartbeater.cc:503] Master 127.5.98.254:39303 was elected leader, sending a full tablet report...
W20260924 15:27:56.552680 9937 tablet.cc:2369] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:56.649098 9804 tablet.cc:2369] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:56.682145 10094 raft_consensus.cc:482] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:56.682283 10094 raft_consensus.cc:504] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.682719 10094 leader_election.cc:288] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516e3e8895394a2e82841cbf0d15fde2 (127.5.98.194:32993), 94f8996d4cbe488c801f1031532de8c9 (127.5.98.195:42793)
I20260924 15:27:56.685775 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.683037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:55607 (local address 127.5.98.194:32993)
0924 15:27:56.683083 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.683085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.683159 (+ 74us) server_negotiation.cc:407] Connection header received
0924 15:27:56.683228 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.683229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.683248 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.683323 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:56.683487 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.684028 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685023 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.685389 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.685504 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.685657 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.685697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:27:56.685782 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.683028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:53353 (local address 127.5.98.195:42793)
0924 15:27:56.683081 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.683083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.683146 (+ 63us) server_negotiation.cc:407] Connection header received
0924 15:27:56.683240 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.683241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.683250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.683315 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:56.683521 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.684012 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685022 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.685386 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685489 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.685498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.685677 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.685717 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:27:56.685835 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.682950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:32993 (local address 127.5.98.193:55607)
0924 15:27:56.683082 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.683093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.683161 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.683331 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.683332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.683334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:56.683412 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.683413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.684122 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.684123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.684922 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.684924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.685507 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.685582 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.685585 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.685628 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.685686 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260924 15:27:56.685863 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.682916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42793 (local address 127.5.98.193:53353)
0924 15:27:56.683079 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.683091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:56.683162 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:56.683323 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:56.683324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:56.683326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:56.683419 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.683421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.684086 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.684087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.684927 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:56.684930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.685511 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.685512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:56.685589 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.685592 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:56.685664 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:27:56.685700 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":22,"thread_start_us":106,"threads_started":1}
I20260924 15:27:56.686286 10023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c6b678a7a346e888cb08fb7ed30228" candidate_uuid: "da1e8b8773944471adaba761bf8bfe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f8996d4cbe488c801f1031532de8c9" is_pre_election: true
I20260924 15:27:56.686290 9890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c6b678a7a346e888cb08fb7ed30228" candidate_uuid: "da1e8b8773944471adaba761bf8bfe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516e3e8895394a2e82841cbf0d15fde2" is_pre_election: true
I20260924 15:27:56.686451 10023 raft_consensus.cc:2454] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1e8b8773944471adaba761bf8bfe47 in term 0.
I20260924 15:27:56.686467 9890 raft_consensus.cc:2454] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da1e8b8773944471adaba761bf8bfe47 in term 0.
I20260924 15:27:56.686674 9691 leader_election.cc:302] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516e3e8895394a2e82841cbf0d15fde2, da1e8b8773944471adaba761bf8bfe47; no voters:
I20260924 15:27:56.686829 10094 raft_consensus.cc:2781] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:56.686892 10094 raft_consensus.cc:482] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:56.686909 10094 raft_consensus.cc:3037] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:56.687604 10094 raft_consensus.cc:504] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.687794 10094 leader_election.cc:288] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [CANDIDATE]: Term 1 election: Requested vote from peers 516e3e8895394a2e82841cbf0d15fde2 (127.5.98.194:32993), 94f8996d4cbe488c801f1031532de8c9 (127.5.98.195:42793)
I20260924 15:27:56.687945 10023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c6b678a7a346e888cb08fb7ed30228" candidate_uuid: "da1e8b8773944471adaba761bf8bfe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f8996d4cbe488c801f1031532de8c9"
I20260924 15:27:56.687953 9890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c6b678a7a346e888cb08fb7ed30228" candidate_uuid: "da1e8b8773944471adaba761bf8bfe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516e3e8895394a2e82841cbf0d15fde2"
I20260924 15:27:56.688043 10023 raft_consensus.cc:3037] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:56.688033 9890 raft_consensus.cc:3037] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:56.688730 9890 raft_consensus.cc:2454] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1e8b8773944471adaba761bf8bfe47 in term 1.
I20260924 15:27:56.688900 10023 raft_consensus.cc:2454] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da1e8b8773944471adaba761bf8bfe47 in term 1.
I20260924 15:27:56.688898 9691 leader_election.cc:302] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516e3e8895394a2e82841cbf0d15fde2, da1e8b8773944471adaba761bf8bfe47; no voters:
I20260924 15:27:56.689007 10094 raft_consensus.cc:2781] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:56.689153 10094 raft_consensus.cc:686] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 1 LEADER]: Becoming Leader. State: Replica: da1e8b8773944471adaba761bf8bfe47, State: Running, Role: LEADER
I20260924 15:27:56.689229 10094 consensus_queue.cc:237] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } }
I20260924 15:27:56.690099 9612 catalog_manager.cc:5495] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 reported cstate change: term changed from 0 to 1, leader changed from <none> to da1e8b8773944471adaba761bf8bfe47 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "da1e8b8773944471adaba761bf8bfe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1e8b8773944471adaba761bf8bfe47" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 } health_report { overall_health: UNKNOWN } } }
W20260924 15:27:56.708767 10071 tablet.cc:2369] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:56.728011 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.725572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.5.98.193:36043)
0924 15:27:56.725650 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.725652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.725663 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:56.725710 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.725711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.725729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.725807 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:56.725964 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.726427 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.727210 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:56.727404 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:56.727479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:56.727674 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:56.727777 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:56.727871 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.727896 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.727918 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:27:56.730055 10023 raft_consensus.cc:1261] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 1 FOLLOWER]: Refusing update from remote peer da1e8b8773944471adaba761bf8bfe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:56.730055 9890 raft_consensus.cc:1261] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 1 FOLLOWER]: Refusing update from remote peer da1e8b8773944471adaba761bf8bfe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:56.730487 10098 consensus_queue.cc:1035] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516e3e8895394a2e82841cbf0d15fde2" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:56.730654 10094 consensus_queue.cc:1035] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f8996d4cbe488c801f1031532de8c9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:56.733268 10101 mvcc.cc:205] Tried to move back new op lower bound from 7332920019882721280 to 7332920019719675904. Current Snapshot: MvccSnapshot[applied={T|T < 7332920019882721280}]
I20260924 15:27:56.733456 10103 mvcc.cc:205] Tried to move back new op lower bound from 7332920019882721280 to 7332920019719675904. Current Snapshot: MvccSnapshot[applied={T|T < 7332920019882721280}]
I20260924 15:27:56.735802 9612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:56.735906 9612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:56.736694 9612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f16b747b8824165b1512c83383d48fb: Sending DeleteTablet for 3 replicas of tablet d4c6b678a7a346e888cb08fb7ed30228
I20260924 15:27:56.737036 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:56 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:43274
I20260924 15:27:56.737084 9870 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:56 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:45100
I20260924 15:27:56.737304 10111 tablet_replica.cc:330] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47: stopping tablet replica
I20260924 15:27:56.737334 10003 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c6b678a7a346e888cb08fb7ed30228 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:56 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:54866
I20260924 15:27:56.737389 10112 tablet_replica.cc:330] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: stopping tablet replica
I20260924 15:27:56.737393 10111 raft_consensus.cc:2229] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:56.737485 10112 raft_consensus.cc:2229] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:56.737516 10111 raft_consensus.cc:2258] T d4c6b678a7a346e888cb08fb7ed30228 P da1e8b8773944471adaba761bf8bfe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:56.737583 10112 raft_consensus.cc:2258] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:56.737736 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9675
I20260924 15:27:56.739641 10113 tablet_replica.cc:330] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: stopping tablet replica
I20260924 15:27:56.739753 10112 ts_tablet_manager.cc:1905] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:56.739809 10113 raft_consensus.cc:2229] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:56.740015 10113 raft_consensus.cc:2258] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:56.740528 10113 ts_tablet_manager.cc:1905] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:56.741662 10112 ts_tablet_manager.cc:1918] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:56.741744 10112 log.cc:1196] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/d4c6b678a7a346e888cb08fb7ed30228
I20260924 15:27:56.742033 10112 ts_tablet_manager.cc:1939] T d4c6b678a7a346e888cb08fb7ed30228 P 516e3e8895394a2e82841cbf0d15fde2: Deleting consensus metadata
I20260924 15:27:56.742403 10113 ts_tablet_manager.cc:1918] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:56.742480 10113 log.cc:1196] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/d4c6b678a7a346e888cb08fb7ed30228
I20260924 15:27:56.742648 9598 catalog_manager.cc:4841] TS 516e3e8895394a2e82841cbf0d15fde2 (127.5.98.194:32993): tablet d4c6b678a7a346e888cb08fb7ed30228 (table test-table [id=e070b9d4cde14b06872e09d366211820]) successfully deleted
I20260924 15:27:56.742838 10113 ts_tablet_manager.cc:1939] T d4c6b678a7a346e888cb08fb7ed30228 P 94f8996d4cbe488c801f1031532de8c9: Deleting consensus metadata
I20260924 15:27:56.743258 9597 catalog_manager.cc:4841] TS 94f8996d4cbe488c801f1031532de8c9 (127.5.98.195:42793): tablet d4c6b678a7a346e888cb08fb7ed30228 (table test-table [id=e070b9d4cde14b06872e09d366211820]) successfully deleted
W20260924 15:27:56.743482 9597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:36043 (error 108)
I20260924 15:27:56.743774 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9808
I20260924 15:27:56.744341 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.743902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36043 (local address 127.0.0.1:43290)
0924 15:27:56.744265 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:56.744314 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
W20260924 15:27:56.744498 9597 catalog_manager.cc:4584] TS da1e8b8773944471adaba761bf8bfe47 (127.5.98.193:36043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4c6b678a7a346e888cb08fb7ed30228: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36043: connect: Connection refused (error 111)
I20260924 15:27:56.751032 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9941
I20260924 15:27:56.755604 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 9582
2026-09-24T15:27:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2117 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T15:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:56Z Disabled control of system clock
I20260924 15:27:56.783319 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43027
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:27:56.866216 10118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:56.866422 10118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:56.866444 10118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:56.867698 10118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:56.867753 10118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:56.867767 10118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:56.867781 10118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:56.867795 10118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:56.869184 10118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:56.869451 10118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:56.869776 10118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:27:56.874837 10118 server_base.cc:1034] running on GCE node
W20260924 15:27:56.874773 10123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.874815 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:56.874800 10124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:56.875285 10118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:56.875521 10118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:56.876717 10118 hybrid_clock.cc:648] HybridClock initialized: now 1790263676876715 us; error 37 us; skew 500 ppm
I20260924 15:27:56.877895 10118 webserver.cc:460] Webserver started at http://127.5.98.254:33297/ using document root <none> and password file <none>
I20260924 15:27:56.878144 10118 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:56.878214 10118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:56.878376 10118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:56.879421 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "c8782ea325254715aacb67e601382f97"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "21d973d530421a44fb2ef99431a1c0cd"
server_key_iv: "8ff298fdd3ea46f4d85acf820cf2ea2b"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.879810 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "c8782ea325254715aacb67e601382f97"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "21d973d530421a44fb2ef99431a1c0cd"
server_key_iv: "8ff298fdd3ea46f4d85acf820cf2ea2b"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.881122 10118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:56.881847 10132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:56.882014 10118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:56.882113 10118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "c8782ea325254715aacb67e601382f97"
format_stamp: "Formatted at 2026-09-24 15:27:56 on dist-test-slave-4g6f"
server_key: "21d973d530421a44fb2ef99431a1c0cd"
server_key_iv: "8ff298fdd3ea46f4d85acf820cf2ea2b"
server_key_version: "encryptionkey@0"
I20260924 15:27:56.882201 10118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:56.885517 10118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:56.885767 10118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:56.885908 10118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:56.889756 10118 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43027
I20260924 15:27:56.889786 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43027 every 8 connection(s)
I20260924 15:27:56.890128 10118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:56.890748 10185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:56.893496 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: Bootstrap starting.
I20260924 15:27:56.894063 10185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:56.894348 10185 log.cc:824] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:56.895026 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: No bootstrap required, opened a new log
I20260924 15:27:56.896646 10185 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } }
I20260924 15:27:56.896782 10185 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:56.896826 10185 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8782ea325254715aacb67e601382f97, State: Initialized, Role: FOLLOWER
I20260924 15:27:56.896977 10185 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } }
I20260924 15:27:56.897038 10185 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:56.897078 10185 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:56.897125 10185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:56.897672 10185 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } }
I20260924 15:27:56.897792 10185 leader_election.cc:302] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8782ea325254715aacb67e601382f97; no voters:
I20260924 15:27:56.897836 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10118
I20260924 15:27:56.897929 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:56.897976 10185 leader_election.cc:288] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:56.898074 10188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:56.898147 5515 external_mini_cluster.cc:1428] Setting key 0bf359ff1a68306ed104d3be1b8beae7
I20260924 15:27:56.898236 10185 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:56.898273 10188 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [term 1 LEADER]: Becoming Leader. State: Replica: c8782ea325254715aacb67e601382f97, State: Running, Role: LEADER
I20260924 15:27:56.898353 10188 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } }
I20260924 15:27:56.898928 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8782ea325254715aacb67e601382f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } } }
I20260924 15:27:56.899010 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:56.899258 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8782ea325254715aacb67e601382f97. Latest consensus state: current_term: 1 leader_uuid: "c8782ea325254715aacb67e601382f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8782ea325254715aacb67e601382f97" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43027 } } }
I20260924 15:27:56.899369 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:56.899700 10199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:56.900529 10199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:56.900955 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:56.898906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.5.98.254:43027)
0924 15:27:56.899107 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:56.899118 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:56.899151 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:27:56.899284 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:56.899291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:56.899338 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:56.899837 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:56.900408 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:56.900412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:56.900412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:56.900416 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:56.900454 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:56.900468 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:56.900537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:56.900724 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 15:27:56.900782 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260924 15:27:56.902033 10199 catalog_manager.cc:1347] Generated new cluster ID: d9d4d9db2b5d4ac597b2b85ea8385cfd
I20260924 15:27:56.902103 10199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:56.908012 10199 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:56.908185 10199 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:56.912936 10199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: Generated new TSK 0
I20260924 15:27:56.913131 10199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:56.921653 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--builtin_ntp_servers=127.5.98.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:27:57.004382 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.004624 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.004662 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.005808 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.005872 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.005913 10209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:57.007341 10209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.007575 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.007907 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.012748 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.012773 10217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.012825 10215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.012924 10209 server_base.cc:1034] running on GCE node
I20260924 15:27:57.013347 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.013600 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.014768 10209 hybrid_clock.cc:648] HybridClock initialized: now 1790263677014755 us; error 33 us; skew 500 ppm
I20260924 15:27:57.016109 10209 webserver.cc:460] Webserver started at http://127.5.98.193:41929/ using document root <none> and password file <none>
I20260924 15:27:57.016352 10209 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.016418 10209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.016556 10209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.017587 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "b83a569da74b4371813a8117c04583c8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d5f03eab382dac4487648766c593a5d1"
server_key_iv: "c2744684c158f49fff7310f35c9cea6a"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.017956 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "b83a569da74b4371813a8117c04583c8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d5f03eab382dac4487648766c593a5d1"
server_key_iv: "c2744684c158f49fff7310f35c9cea6a"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.019287 10209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:57.020087 10223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.020255 10209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.020354 10209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "b83a569da74b4371813a8117c04583c8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d5f03eab382dac4487648766c593a5d1"
server_key_iv: "c2744684c158f49fff7310f35c9cea6a"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.020440 10209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.023675 10209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.023937 10209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.024076 10209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.024294 10209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:57.024600 10209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:57.024653 10209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.024694 10209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:57.024726 10209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.030483 10209 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:33359
I20260924 15:27:57.030534 10336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:33359 every 8 connection(s)
I20260924 15:27:57.030829 10209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:57.033862 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.031548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:38003 (local address 127.5.98.254:43027)
0924 15:27:57.031635 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.031638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.031843 (+ 205us) server_negotiation.cc:407] Connection header received
0924 15:27:57.032352 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.032353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.032360 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.032426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.033625 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.033626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.033627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.033631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.033647 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.033651 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.033701 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.033772 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.033788 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:57.034058 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.031459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43027 (local address 127.5.98.193:38003)
0924 15:27:57.031717 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.031760 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.032242 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.032494 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.032498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.032508 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.033468 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.033468 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.033496 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.033523 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.033697 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.033702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.033705 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.033780 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.033839 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
I20260924 15:27:57.034528 10337 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43027
I20260924 15:27:57.034607 10337 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:57.034773 10337 heartbeater.cc:511] Master 127.5.98.254:43027 requested a full tablet report, sending...
I20260924 15:27:57.035270 10149 ts_manager.cc:194] Registered new tserver with Master: b83a569da74b4371813a8117c04583c8 (127.5.98.193:33359)
I20260924 15:27:57.036404 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10209
I20260924 15:27:57.036527 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:57.036705 5515 external_mini_cluster.cc:1428] Setting key ffda14811207866ead4ead4cefb98ffb
I20260924 15:27:57.037587 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--builtin_ntp_servers=127.5.98.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:27:57.117255 10340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.117473 10340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.117507 10340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.118646 10340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.118707 10340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.118752 10340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:27:57.120114 10340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.120373 10340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.120661 10340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.125804 10348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.125772 10345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.125859 10346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.126847 10340 server_base.cc:1034] running on GCE node
I20260924 15:27:57.127100 10340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.127351 10340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.128512 10340 hybrid_clock.cc:648] HybridClock initialized: now 1790263677128530 us; error 65 us; skew 500 ppm
I20260924 15:27:57.129777 10340 webserver.cc:460] Webserver started at http://127.5.98.194:41735/ using document root <none> and password file <none>
I20260924 15:27:57.130029 10340 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.130098 10340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.130266 10340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.131409 10340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "33dac61bf08949a3a6744f5e86af21ef"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "68b986c966b5aecf4431727714578da9"
server_key_iv: "f34acd9e992f3592ac1e7426f2e5c300"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.131862 10340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "33dac61bf08949a3a6744f5e86af21ef"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "68b986c966b5aecf4431727714578da9"
server_key_iv: "f34acd9e992f3592ac1e7426f2e5c300"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.133172 10340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:57.133940 10354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.134124 10340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:57.134222 10340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "33dac61bf08949a3a6744f5e86af21ef"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "68b986c966b5aecf4431727714578da9"
server_key_iv: "f34acd9e992f3592ac1e7426f2e5c300"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.134307 10340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.137656 10340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.137916 10340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.138059 10340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.138267 10340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:57.138566 10340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:57.138626 10340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.138666 10340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:57.138695 10340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.144325 10340 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:43659
I20260924 15:27:57.144363 10467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:43659 every 8 connection(s)
I20260924 15:27:57.144656 10340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:57.147210 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.145277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:38889 (local address 127.5.98.254:43027)
0924 15:27:57.145326 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.145328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.145564 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:27:57.145981 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.145982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.145987 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.146033 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.147038 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.147039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.147039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.147043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.147058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.147061 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.147106 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.147151 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.147161 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:57.147390 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.145289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43027 (local address 127.5.98.194:38889)
0924 15:27:57.145450 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.145480 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.145903 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.146087 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.146091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.146100 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.146915 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.146915 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.146941 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.146965 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.147105 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.147107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.147109 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.147156 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.147200 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260924 15:27:57.147871 10468 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43027
I20260924 15:27:57.147956 10468 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:57.148095 10468 heartbeater.cc:511] Master 127.5.98.254:43027 requested a full tablet report, sending...
I20260924 15:27:57.148485 10149 ts_manager.cc:194] Registered new tserver with Master: 33dac61bf08949a3a6744f5e86af21ef (127.5.98.194:43659)
I20260924 15:27:57.152015 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10340
I20260924 15:27:57.152102 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:57.152240 5515 external_mini_cluster.cc:1428] Setting key 4293ace34c9f84e56e1b585d3e7da783
I20260924 15:27:57.153069 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--builtin_ntp_servers=127.5.98.212:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:27:57.234033 10471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.234221 10471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.234242 10471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.235324 10471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.235358 10471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.235383 10471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:27:57.236653 10471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.236907 10471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.237213 10471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.242161 10479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.242254 10476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.242425 10471 server_base.cc:1034] running on GCE node
W20260924 15:27:57.242170 10477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.242717 10471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.242990 10471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.244146 10471 hybrid_clock.cc:648] HybridClock initialized: now 1790263677244128 us; error 39 us; skew 500 ppm
I20260924 15:27:57.257506 10471 webserver.cc:460] Webserver started at http://127.5.98.195:44555/ using document root <none> and password file <none>
I20260924 15:27:57.257790 10471 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.257874 10471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.258033 10471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.259152 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "b6a45b959492405f88fec14cdaaea948"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "4b0a65ada4c177e7eee4745a1a9f02fc"
server_key_iv: "dc5bfd9e393eee90e446d57e0f6e2c25"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.259526 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "b6a45b959492405f88fec14cdaaea948"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "4b0a65ada4c177e7eee4745a1a9f02fc"
server_key_iv: "dc5bfd9e393eee90e446d57e0f6e2c25"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.260996 10471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:57.261785 10485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.261986 10471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:57.262090 10471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "b6a45b959492405f88fec14cdaaea948"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "4b0a65ada4c177e7eee4745a1a9f02fc"
server_key_iv: "dc5bfd9e393eee90e446d57e0f6e2c25"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.262176 10471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.265607 10471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.265877 10471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.266021 10471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.266227 10471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:57.266530 10471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:57.266583 10471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.266633 10471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:57.266660 10471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.272423 10471 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46475
I20260924 15:27:57.272464 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46475 every 8 connection(s)
I20260924 15:27:57.272802 10471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:57.275791 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.273535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:55007 (local address 127.5.98.254:43027)
0924 15:27:57.273602 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.273604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.273849 (+ 245us) server_negotiation.cc:407] Connection header received
0924 15:27:57.274282 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.274284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.274290 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.274341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.275505 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.275506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.275506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.275510 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.275525 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.275529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.275590 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.275705 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.275721 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:57.275990 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.273548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43027 (local address 127.5.98.195:55007)
0924 15:27:57.273731 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.273762 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.274201 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.274392 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.274395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.274403 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.275321 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.275321 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.275349 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.275376 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.275600 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.275604 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.275607 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.275713 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.275774 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260924 15:27:57.276499 10599 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43027
I20260924 15:27:57.276592 10599 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:57.276718 10599 heartbeater.cc:511] Master 127.5.98.254:43027 requested a full tablet report, sending...
I20260924 15:27:57.277051 10149 ts_manager.cc:194] Registered new tserver with Master: b6a45b959492405f88fec14cdaaea948 (127.5.98.195:46475)
I20260924 15:27:57.277700 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10471
I20260924 15:27:57.277787 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:57.277928 5515 external_mini_cluster.cc:1428] Setting key 61204f878eeb5dcdc4ce5e7030b528d6
I20260924 15:27:57.278769 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:57.281211 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.280057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.5.98.254:43027)
0924 15:27:57.280138 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.280140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.280243 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:27:57.280333 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.280334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.280340 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.280390 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.280990 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.280991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.280992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.280996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.281012 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.281015 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.281071 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.281127 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.281143 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 15:27:57.282857 10149 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:57.284497 10149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:57.288365 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46475 (local address 127.0.0.1:60842)
0924 15:27:57.287221 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.287230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.287267 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.287439 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.287440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.287443 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.287911 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.287912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.287927 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.287950 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.288195 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.288202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.288206 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.288267 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.288281 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260924 15:27:57.288633 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.5.98.195:46475)
0924 15:27:57.286916 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.286920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.287323 (+ 403us) server_negotiation.cc:407] Connection header received
0924 15:27:57.287348 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.287350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.287359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.287425 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.288044 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.288050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.288050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.288054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.288084 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.288109 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.288192 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.288539 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.288557 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260924 15:27:57.288635 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:43659 (local address 127.0.0.1:57692)
0924 15:27:57.287044 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.287054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.287121 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.287868 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.287870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.287873 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.288263 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.288264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.288270 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.288282 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.288489 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.288490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.288491 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.288548 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.288562 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":92,"threads_started":1}
I20260924 15:27:57.288647 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.5.98.194:43659)
0924 15:27:57.286985 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.286990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.287718 (+ 728us) server_negotiation.cc:407] Connection header received
0924 15:27:57.287753 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.287756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.287767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.287853 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.288364 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.288372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.288372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.288377 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.288406 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.288435 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.288493 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.288557 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.288572 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260924 15:27:57.288784 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33359 (local address 127.0.0.1:49124)
0924 15:27:57.286675 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.286710 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.286762 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.287149 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.287155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.287160 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.287921 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.287922 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.287929 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.287954 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.288631 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.288633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.288633 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.288686 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.288700 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260924 15:27:57.289108 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.286803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.5.98.193:33359)
0924 15:27:57.286986 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.286990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.287000 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:57.287029 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.287031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.287041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.287142 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.288275 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.288282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.288282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.288285 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.288343 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.288367 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.288443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.289008 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.289025 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260924 15:27:57.289495 10402 tablet_service.cc:1467] Processing CreateTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 (DEFAULT_TABLE table=test-table [id=52c9576a6fb847a2b3725bde835c8229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:57.289857 10402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49515b01a17f49f48f70c7ad7bf81877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.289798 10271 tablet_service.cc:1467] Processing CreateTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 (DEFAULT_TABLE table=test-table [id=52c9576a6fb847a2b3725bde835c8229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:57.290124 10271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49515b01a17f49f48f70c7ad7bf81877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.291985 10533 tablet_service.cc:1467] Processing CreateTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 (DEFAULT_TABLE table=test-table [id=52c9576a6fb847a2b3725bde835c8229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:57.292281 10533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49515b01a17f49f48f70c7ad7bf81877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.293221 10613 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Bootstrap starting.
I20260924 15:27:57.293438 10614 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: Bootstrap starting.
I20260924 15:27:57.294149 10614 tablet_bootstrap.cc:654] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.294474 10614 log.cc:824] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.295184 10614 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: No bootstrap required, opened a new log
I20260924 15:27:57.295315 10614 ts_tablet_manager.cc:1397] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:57.295394 10613 tablet_bootstrap.cc:654] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.295411 10616 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Bootstrap starting.
I20260924 15:27:57.295837 10613 log.cc:824] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.296120 10616 tablet_bootstrap.cc:654] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.296489 10616 log.cc:824] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.296523 10613 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: No bootstrap required, opened a new log
I20260924 15:27:57.296566 10613 ts_tablet_manager.cc:1397] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:27:57.297290 10616 tablet_bootstrap.cc:492] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: No bootstrap required, opened a new log
I20260924 15:27:57.297363 10616 ts_tablet_manager.cc:1397] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:57.298097 10614 raft_consensus.cc:348] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.298295 10614 raft_consensus.cc:374] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.298354 10614 raft_consensus.cc:729] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83a569da74b4371813a8117c04583c8, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.298497 10614 consensus_queue.cc:260] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.298756 10614 ts_tablet_manager.cc:1428] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:57.298877 10337 heartbeater.cc:503] Master 127.5.98.254:43027 was elected leader, sending a full tablet report...
I20260924 15:27:57.299073 10613 raft_consensus.cc:348] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.299256 10613 raft_consensus.cc:374] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.299294 10613 raft_consensus.cc:729] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dac61bf08949a3a6744f5e86af21ef, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.299443 10613 consensus_queue.cc:260] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.299736 10613 ts_tablet_manager.cc:1428] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:57.299751 10468 heartbeater.cc:503] Master 127.5.98.254:43027 was elected leader, sending a full tablet report...
I20260924 15:27:57.300546 10616 raft_consensus.cc:348] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.300696 10616 raft_consensus.cc:374] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.300738 10616 raft_consensus.cc:729] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a45b959492405f88fec14cdaaea948, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.300854 10616 consensus_queue.cc:260] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.301124 10616 ts_tablet_manager.cc:1428] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 15:27:57.301164 10599 heartbeater.cc:503] Master 127.5.98.254:43027 was elected leader, sending a full tablet report...
I20260924 15:27:57.304093 10621 raft_consensus.cc:482] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:57.304157 10621 raft_consensus.cc:504] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.304464 10621 leader_election.cc:288] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83a569da74b4371813a8117c04583c8 (127.5.98.193:33359), 33dac61bf08949a3a6744f5e86af21ef (127.5.98.194:43659)
I20260924 15:27:57.305524 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.304615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33359 (local address 127.5.98.195:53861)
0924 15:27:57.304675 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.304687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.304731 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.304853 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.304854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.304856 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.305227 (+ 371us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.305228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.305235 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.305246 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.305398 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.305398 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.305400 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.305451 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.305465 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:27:57.305524 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.304650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:53861 (local address 127.5.98.193:33359)
0924 15:27:57.304695 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.304697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.304720 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:27:57.304789 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.304790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.304794 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.304850 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.305317 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.305318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.305319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.305323 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.305338 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.305342 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.305398 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.305453 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.305466 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:27:57.305938 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49515b01a17f49f48f70c7ad7bf81877" candidate_uuid: "b6a45b959492405f88fec14cdaaea948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a569da74b4371813a8117c04583c8" is_pre_election: true
I20260924 15:27:57.306071 10291 raft_consensus.cc:2454] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a45b959492405f88fec14cdaaea948 in term 0.
I20260924 15:27:57.306128 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.304668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:42525 (local address 127.5.98.194:43659)
0924 15:27:57.304707 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.304709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.304924 (+ 215us) server_negotiation.cc:407] Connection header received
0924 15:27:57.305231 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.305232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.305236 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.305306 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.305896 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.305897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.305898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.305903 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.305918 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.305922 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.305978 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.306063 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.306080 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:57.306150 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.304616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:43659 (local address 127.5.98.195:42525)
0924 15:27:57.304849 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.304859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.304932 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.305312 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.305313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.305315 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:57.305771 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:57.305772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:57.305781 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:57.305796 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:57.306006 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:57.306007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:57.306007 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 15:27:57.306060 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:27:57.306073 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":103,"thread_start_us":89,"threads_started":1}
I20260924 15:27:57.306358 10487 leader_election.cc:302] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a45b959492405f88fec14cdaaea948, b83a569da74b4371813a8117c04583c8; no voters:
I20260924 15:27:57.306445 10422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49515b01a17f49f48f70c7ad7bf81877" candidate_uuid: "b6a45b959492405f88fec14cdaaea948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dac61bf08949a3a6744f5e86af21ef" is_pre_election: true
I20260924 15:27:57.306490 10621 raft_consensus.cc:2781] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:57.306535 10621 raft_consensus.cc:482] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:57.306562 10621 raft_consensus.cc:3037] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.306568 10422 raft_consensus.cc:2454] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a45b959492405f88fec14cdaaea948 in term 0.
I20260924 15:27:57.307183 10621 raft_consensus.cc:504] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.307322 10621 leader_election.cc:288] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [CANDIDATE]: Term 1 election: Requested vote from peers b83a569da74b4371813a8117c04583c8 (127.5.98.193:33359), 33dac61bf08949a3a6744f5e86af21ef (127.5.98.194:43659)
I20260924 15:27:57.307446 10291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49515b01a17f49f48f70c7ad7bf81877" candidate_uuid: "b6a45b959492405f88fec14cdaaea948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83a569da74b4371813a8117c04583c8"
I20260924 15:27:57.307478 10422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49515b01a17f49f48f70c7ad7bf81877" candidate_uuid: "b6a45b959492405f88fec14cdaaea948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dac61bf08949a3a6744f5e86af21ef"
I20260924 15:27:57.307535 10422 raft_consensus.cc:3037] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.307528 10291 raft_consensus.cc:3037] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.308375 10422 raft_consensus.cc:2454] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a45b959492405f88fec14cdaaea948 in term 1.
I20260924 15:27:57.308375 10291 raft_consensus.cc:2454] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a45b959492405f88fec14cdaaea948 in term 1.
I20260924 15:27:57.308524 10487 leader_election.cc:302] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a45b959492405f88fec14cdaaea948, b83a569da74b4371813a8117c04583c8; no voters:
I20260924 15:27:57.308634 10621 raft_consensus.cc:2781] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:57.308790 10621 raft_consensus.cc:686] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 1 LEADER]: Becoming Leader. State: Replica: b6a45b959492405f88fec14cdaaea948, State: Running, Role: LEADER
I20260924 15:27:57.308902 10621 consensus_queue.cc:237] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } }
I20260924 15:27:57.309602 10148 catalog_manager.cc:5495] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6a45b959492405f88fec14cdaaea948 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "b6a45b959492405f88fec14cdaaea948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a45b959492405f88fec14cdaaea948" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46475 } health_report { overall_health: HEALTHY } } }
I20260924 15:27:57.313503 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.312686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.5.98.195:46475)
0924 15:27:57.312741 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.312743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.312751 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:57.312782 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.312783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.312793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.312843 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.313278 (+ 435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.313279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.313280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.313284 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.313296 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.313300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.313355 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.313425 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.313440 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:27:57.314914 10422 raft_consensus.cc:1261] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 1 FOLLOWER]: Refusing update from remote peer b6a45b959492405f88fec14cdaaea948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:57.314934 10291 raft_consensus.cc:1261] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 1 FOLLOWER]: Refusing update from remote peer b6a45b959492405f88fec14cdaaea948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:57.315253 10623 consensus_queue.cc:1035] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dac61bf08949a3a6744f5e86af21ef" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:57.315366 10621 consensus_queue.cc:1035] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83a569da74b4371813a8117c04583c8" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:57.317736 10628 mvcc.cc:205] Tried to move back new op lower bound from 7332920022279852032 to 7332920022258012160. Current Snapshot: MvccSnapshot[applied={T|T < 7332920022279852032}]
I20260924 15:27:57.317767 10627 mvcc.cc:205] Tried to move back new op lower bound from 7332920022279852032 to 7332920022258012160. Current Snapshot: MvccSnapshot[applied={T|T < 7332920022279852032}]
I20260924 15:27:57.319635 10148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:57.319768 10148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:57.320505 10148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8782ea325254715aacb67e601382f97: Sending DeleteTablet for 3 replicas of tablet 49515b01a17f49f48f70c7ad7bf81877
I20260924 15:27:57.320829 10271 tablet_service.cc:1514] Processing DeleteTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:57 UTC) from {username='slave'} at 127.0.0.1:49124
I20260924 15:27:57.320834 10533 tablet_service.cc:1514] Processing DeleteTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:57 UTC) from {username='slave'} at 127.0.0.1:60842
I20260924 15:27:57.320868 10402 tablet_service.cc:1514] Processing DeleteTablet for tablet 49515b01a17f49f48f70c7ad7bf81877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:57 UTC) from {username='slave'} at 127.0.0.1:57692
I20260924 15:27:57.321064 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10209
I20260924 15:27:57.321126 10636 tablet_replica.cc:330] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8: stopping tablet replica
I20260924 15:27:57.321211 10636 raft_consensus.cc:2229] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:57.321297 10636 raft_consensus.cc:2258] T 49515b01a17f49f48f70c7ad7bf81877 P b83a569da74b4371813a8117c04583c8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 15:27:57.326470 10134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:33359 (error 108)
I20260924 15:27:57.326560 10638 tablet_replica.cc:330] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: stopping tablet replica
I20260924 15:27:57.326705 10638 raft_consensus.cc:2229] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:57.326896 10638 raft_consensus.cc:2258] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:57.327179 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10340
I20260924 15:27:57.327211 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.327067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33359 (local address 127.0.0.1:49140)
0924 15:27:57.327128 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.327171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 15:27:57.327314 10134 catalog_manager.cc:4584] TS b83a569da74b4371813a8117c04583c8 (127.5.98.193:33359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49515b01a17f49f48f70c7ad7bf81877: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33359: connect: Connection refused (error 111)
I20260924 15:27:57.327396 10638 ts_tablet_manager.cc:1905] T 49515b01a17f49f48f70c7ad7bf81877 P 33dac61bf08949a3a6744f5e86af21ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:57.327833 10637 tablet_replica.cc:330] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: stopping tablet replica
I20260924 15:27:57.327929 10637 raft_consensus.cc:2229] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:57.328029 10637 raft_consensus.cc:2258] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:57.328794 10637 ts_tablet_manager.cc:1905] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:57.330444 10637 ts_tablet_manager.cc:1918] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:57.330519 10637 log.cc:1196] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/49515b01a17f49f48f70c7ad7bf81877
I20260924 15:27:57.330930 10637 ts_tablet_manager.cc:1939] T 49515b01a17f49f48f70c7ad7bf81877 P b6a45b959492405f88fec14cdaaea948: Deleting consensus metadata
I20260924 15:27:57.331329 10136 catalog_manager.cc:4841] TS b6a45b959492405f88fec14cdaaea948 (127.5.98.195:46475): tablet 49515b01a17f49f48f70c7ad7bf81877 (table test-table [id=52c9576a6fb847a2b3725bde835c8229]) successfully deleted
I20260924 15:27:57.332963 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10471
I20260924 15:27:57.333598 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.333085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:43659 (local address 127.0.0.1:57706)
0924 15:27:57.333538 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.333571 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.194:43659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
I20260924 15:27:57.337929 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10118
2026-09-24T15:27:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (581 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 15:27:57.357605 5515 mini_oidc.cc:152] Starting JWKS server
I20260924 15:27:57.359840 5515 webserver.cc:460] Webserver started at https://127.0.0.1:35539/ using document root <none> and password file <none>
I20260924 15:27:57.359959 5515 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:27:57.360152 5515 webserver.cc:460] Webserver started at http://127.0.0.1:37873/ using document root <none> and password file <none>
2026-09-24T15:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:57Z Disabled control of system clock
I20260924 15:27:57.367883 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44671
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:44671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/testchainca.cert with env {}
W20260924 15:27:57.447870 10645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.448077 10645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.448098 10645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.449296 10645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:57.449339 10645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.449352 10645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.449406 10645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:57.449433 10645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:27:57.449450 10645 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:27:57.449472 10645 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35539/jwks.json
I20260924 15:27:57.451040 10645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:44671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.451297 10645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.451530 10645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.456385 10651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.456347 10653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.456398 10650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.456516 10645 server_base.cc:1034] running on GCE node
I20260924 15:27:57.456862 10645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.457191 10645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.458388 10645 hybrid_clock.cc:648] HybridClock initialized: now 1790263677458375 us; error 45 us; skew 500 ppm
I20260924 15:27:57.459885 10645 webserver.cc:460] Webserver started at http://127.5.98.254:33931/ using document root <none> and password file <none>
I20260924 15:27:57.460150 10645 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.460228 10645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.460388 10645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.461539 10645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "4ee1d5c486da465e9709a70f1a7cd358"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d74d150049ee23b47a37fe6c371b97ae"
server_key_iv: "b831b975076772abb2bca5e842523996"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.461956 10645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "4ee1d5c486da465e9709a70f1a7cd358"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d74d150049ee23b47a37fe6c371b97ae"
server_key_iv: "b831b975076772abb2bca5e842523996"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.463343 10645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:57.464231 10659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.464411 10645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.464550 10645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "4ee1d5c486da465e9709a70f1a7cd358"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "d74d150049ee23b47a37fe6c371b97ae"
server_key_iv: "b831b975076772abb2bca5e842523996"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.464644 10645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.479353 10645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.479782 10645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.479964 10645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.484222 10645 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:44671
I20260924 15:27:57.484252 10711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:44671 every 8 connection(s)
I20260924 15:27:57.484639 10645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:57.485579 10712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.487854 10712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358: Bootstrap starting.
I20260924 15:27:57.488394 10712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.488634 10712 log.cc:824] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.489286 10712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358: No bootstrap required, opened a new log
I20260924 15:27:57.490749 10712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } }
I20260924 15:27:57.490885 10712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.490908 10712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee1d5c486da465e9709a70f1a7cd358, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.491041 10712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } }
I20260924 15:27:57.491129 10712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:57.491151 10712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:57.491180 10712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.491626 10712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } }
I20260924 15:27:57.491775 10712 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee1d5c486da465e9709a70f1a7cd358; no voters:
I20260924 15:27:57.491894 10712 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:57.491974 10715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:57.492173 10712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:57.492157 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10645
I20260924 15:27:57.492257 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:57.492190 10715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee1d5c486da465e9709a70f1a7cd358, State: Running, Role: LEADER
I20260924 15:27:57.492384 10715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } }
I20260924 15:27:57.492457 5515 external_mini_cluster.cc:1428] Setting key fd673f2a63c4099e501dd4461d31bd84
I20260924 15:27:57.493216 10716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee1d5c486da465e9709a70f1a7cd358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } } }
I20260924 15:27:57.493299 10716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.493463 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee1d5c486da465e9709a70f1a7cd358. Latest consensus state: current_term: 1 leader_uuid: "4ee1d5c486da465e9709a70f1a7cd358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee1d5c486da465e9709a70f1a7cd358" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44671 } } }
I20260924 15:27:57.493534 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.493539 10729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:57.494204 10729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:57.495942 10729 catalog_manager.cc:1347] Generated new cluster ID: c9ffe43a8dd74d84997c2d63c64e6227
I20260924 15:27:57.495995 10729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:57.498483 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.492986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.5.98.254:44671)
0924 15:27:57.493672 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.493682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.493867 (+ 185us) server_negotiation.cc:407] Connection header received
0924 15:27:57.494001 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.494007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.494062 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.494207 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.494459 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.495638 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.496565 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.497016 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.497158 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.497849 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.497853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.497853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.498028 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.498061 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.498071 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.498150 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.498243 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.498325 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":80,"threads_started":1}
I20260924 15:27:57.504565 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:57.518289 10729 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:57.518884 10729 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:57.525645 10729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee1d5c486da465e9709a70f1a7cd358: Generated new TSK 0
I20260924 15:27:57.525826 10729 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:27:57.557337 10742 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 15:27:57.557510 10742 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:27:57.557725 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.534413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.5.98.254:44671)
0924 15:27:57.534453 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.534455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.534610 (+ 155us) server_negotiation.cc:407] Connection header received
0924 15:27:57.534715 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.534716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.534729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.534810 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:27:57.534970 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.535469 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.536319 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.536514 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.536587 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.536810 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:27:57.557342 (+ 20532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:27:57.557654 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:27:57.557785 10725 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 15:27:57.558043 10741 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:57.534353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44671 (local address 127.0.0.1:59888)
0924 15:27:57.534536 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.534546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.534634 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.534821 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.534822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.534833 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:27:57.534894 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.534896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.535556 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.535557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:57.536230 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.536232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.536599 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.536601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:57.536667 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.536694 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:27:57.557486 (+ 20792us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:27:57.557616 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.98.254:44671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.98.254:44671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 15:27:57.558632 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10645
2026-09-24T15:27:57Z chronyd exiting
I20260924 15:27:57.594489 5515 test_util.cc:175] -----------------------------------------------
I20260924 15:27:57.594568 5515 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263655576751-5515-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 15:27:57.595289 5515 mini_oidc.cc:152] Starting JWKS server
I20260924 15:27:57.596028 5515 webserver.cc:460] Webserver started at https://127.0.0.1:37533/ using document root <none> and password file <none>
I20260924 15:27:57.596069 5515 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:27:57.596180 5515 webserver.cc:460] Webserver started at http://127.0.0.1:43357/ using document root <none> and password file <none>
2026-09-24T15:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:57Z Disabled control of system clock
I20260924 15:27:57.603965 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43723
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43723
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37533/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/testchainca.cert with env {}
W20260924 15:27:57.683115 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.683313 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.683334 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.684454 10748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:57.684491 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.684504 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.684517 10748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:57.684530 10748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:27:57.684541 10748 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:27:57.684552 10748 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37533/jwks.json
I20260924 15:27:57.686017 10748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37533/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.686220 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.686389 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.691107 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.691175 10756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.691124 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.691573 10748 server_base.cc:1034] running on GCE node
I20260924 15:27:57.691834 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.692116 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.693316 10748 hybrid_clock.cc:648] HybridClock initialized: now 1790263677693305 us; error 41 us; skew 500 ppm
I20260924 15:27:57.694690 10748 webserver.cc:460] Webserver started at http://127.5.98.254:41945/ using document root <none> and password file <none>
I20260924 15:27:57.694983 10748 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.695061 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.695237 10748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.696483 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "32a31f20ddd0475c8c0feef56bb2cf5c"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "51452b90cf597b20dc832ee362eec0fc"
server_key_iv: "de0e02ce26e3160c9e267ddfec00e410"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.696879 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "32a31f20ddd0475c8c0feef56bb2cf5c"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "51452b90cf597b20dc832ee362eec0fc"
server_key_iv: "de0e02ce26e3160c9e267ddfec00e410"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.698269 10748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:57.699031 10762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.699195 10748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:57.699306 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "32a31f20ddd0475c8c0feef56bb2cf5c"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "51452b90cf597b20dc832ee362eec0fc"
server_key_iv: "de0e02ce26e3160c9e267ddfec00e410"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.699393 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.719026 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.719353 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.719527 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.723631 10748 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43723
I20260924 15:27:57.723718 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43723 every 8 connection(s)
I20260924 15:27:57.724093 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:57.725035 10815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.727209 10815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c: Bootstrap starting.
I20260924 15:27:57.727746 10815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.727898 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10748
I20260924 15:27:57.728011 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:57.728099 10815 log.cc:824] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.728240 5515 external_mini_cluster.cc:1428] Setting key 7b6f01bae573510af6a904c948c4ead6
I20260924 15:27:57.728864 10815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c: No bootstrap required, opened a new log
I20260924 15:27:57.730495 10815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } }
I20260924 15:27:57.730629 10815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.730652 10815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a31f20ddd0475c8c0feef56bb2cf5c, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.730801 10815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } }
I20260924 15:27:57.730917 10815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:57.730989 10815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:57.731043 10815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.731853 10815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } }
I20260924 15:27:57.731963 10815 leader_election.cc:302] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a31f20ddd0475c8c0feef56bb2cf5c; no voters:
I20260924 15:27:57.732115 10815 leader_election.cc:288] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:57.732165 10820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:57.732379 10820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [term 1 LEADER]: Becoming Leader. State: Replica: 32a31f20ddd0475c8c0feef56bb2cf5c, State: Running, Role: LEADER
I20260924 15:27:57.732394 10815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:57.732578 10820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } }
I20260924 15:27:57.732949 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } } }
I20260924 15:27:57.733035 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.733260 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a31f20ddd0475c8c0feef56bb2cf5c. Latest consensus state: current_term: 1 leader_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a31f20ddd0475c8c0feef56bb2cf5c" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43723 } } }
I20260924 15:27:57.733326 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.733577 10825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:57.734179 10825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:57.736577 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.728763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.5.98.254:43723)
0924 15:27:57.729016 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.729024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.729047 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:27:57.730498 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.730506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.730584 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.730768 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.731011 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.732803 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.734018 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.734667 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.735337 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.735615 (+ 278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.735618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.735618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.735887 (+ 269us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.735928 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.735942 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.736009 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.736155 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.736367 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260924 15:27:57.736817 10825 catalog_manager.cc:1347] Generated new cluster ID: bf91e1cd4c944500811324eaaa4abb4b
I20260924 15:27:57.736886 10825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:57.738166 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:57.754822 10825 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:57.755358 10825 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:57.761026 10825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a31f20ddd0475c8c0feef56bb2cf5c: Generated new TSK 0
I20260924 15:27:57.761225 10825 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:27:57.787063 10845 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 15:27:57.787166 10845 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 15:27:57.787582 10844 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:57.767423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43723 (local address 127.0.0.1:37466)
0924 15:27:57.767583 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.767610 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.767700 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.767872 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.767873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.767875 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:27:57.767947 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.767953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.768606 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.768607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:57.769222 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.769224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.769604 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.769605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:57.769664 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.769669 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:27:57.787416 (+ 17747us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:27:57.787510 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.98.254:43723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260924 15:27:57.787583 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.767479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.5.98.254:43723)
0924 15:27:57.767554 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.767556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.767673 (+ 117us) server_negotiation.cc:407] Connection header received
0924 15:27:57.767786 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.767787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.767799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.767867 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:27:57.768021 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.768509 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.769323 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.769523 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.769595 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.769777 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:27:57.787279 (+ 17502us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:27:57.787476 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 15:27:57.787644 10819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 15:27:57.787935 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10748
2026-09-24T15:27:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 15:27:57.826112 5515 mini_oidc.cc:152] Starting JWKS server
I20260924 15:27:57.826831 5515 webserver.cc:460] Webserver started at https://127.0.0.1:40223/ using document root <none> and password file <none>
I20260924 15:27:57.826897 5515 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:27:57.827069 5515 webserver.cc:460] Webserver started at http://127.0.0.1:40117/ using document root <none> and password file <none>
2026-09-24T15:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:57Z Disabled control of system clock
I20260924 15:27:57.834612 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41141
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41141
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40223/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:27:57.912917 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:57.913165 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:57.913203 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:57.914351 10851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:57.914415 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:57.914438 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:57.914458 10851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:57.914489 10851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:27:57.914508 10851 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:27:57.914529 10851 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40223/jwks.json
I20260924 15:27:57.915927 10851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40223/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:57.916200 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:57.916477 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:57.921187 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.921238 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:57.921306 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:57.921773 10851 server_base.cc:1034] running on GCE node
I20260924 15:27:57.921968 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:57.922240 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:57.923390 10851 hybrid_clock.cc:648] HybridClock initialized: now 1790263677923379 us; error 31 us; skew 500 ppm
I20260924 15:27:57.924635 10851 webserver.cc:460] Webserver started at http://127.5.98.254:34577/ using document root <none> and password file <none>
I20260924 15:27:57.924816 10851 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:57.924855 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:57.924942 10851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:57.925973 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "786a3af2efc64a37ac98f8366d20e0e8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "8ff7bb45d26d2e286440bd176c9f89f7"
server_key_iv: "fa8c5e7ea4249ee1673418d5aeec5f19"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.926280 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "786a3af2efc64a37ac98f8366d20e0e8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "8ff7bb45d26d2e286440bd176c9f89f7"
server_key_iv: "fa8c5e7ea4249ee1673418d5aeec5f19"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.927465 10851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:57.928253 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.928400 10851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:57.928484 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "786a3af2efc64a37ac98f8366d20e0e8"
format_stamp: "Formatted at 2026-09-24 15:27:57 on dist-test-slave-4g6f"
server_key: "8ff7bb45d26d2e286440bd176c9f89f7"
server_key_iv: "fa8c5e7ea4249ee1673418d5aeec5f19"
server_key_version: "encryptionkey@0"
I20260924 15:27:57.928539 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:57.948879 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:57.949199 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:57.949304 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:57.952971 10851 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:41141
I20260924 15:27:57.953027 10917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:41141 every 8 connection(s)
I20260924 15:27:57.953347 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:57.954051 10918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:57.956841 10918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8: Bootstrap starting.
I20260924 15:27:57.957413 10918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:57.957711 10918 log.cc:824] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:57.958379 10918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8: No bootstrap required, opened a new log
I20260924 15:27:57.959183 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10851
I20260924 15:27:57.959267 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:57.959442 5515 external_mini_cluster.cc:1428] Setting key a5dd916ff84704024e6a973d46b5a3dd
I20260924 15:27:57.960454 10918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } }
I20260924 15:27:57.960638 10918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:57.960676 10918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786a3af2efc64a37ac98f8366d20e0e8, State: Initialized, Role: FOLLOWER
I20260924 15:27:57.960856 10918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } }
I20260924 15:27:57.960974 10918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:57.961007 10918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:57.961057 10918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:57.961771 10918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } }
I20260924 15:27:57.961881 10918 leader_election.cc:302] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 786a3af2efc64a37ac98f8366d20e0e8; no voters:
I20260924 15:27:57.962028 10918 leader_election.cc:288] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:57.962220 10923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:57.962366 10923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [term 1 LEADER]: Becoming Leader. State: Replica: 786a3af2efc64a37ac98f8366d20e0e8, State: Running, Role: LEADER
I20260924 15:27:57.962440 10918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:57.962495 10923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } }
I20260924 15:27:57.962858 10923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786a3af2efc64a37ac98f8366d20e0e8. Latest consensus state: current_term: 1 leader_uuid: "786a3af2efc64a37ac98f8366d20e0e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } } }
I20260924 15:27:57.962925 10925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "786a3af2efc64a37ac98f8366d20e0e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786a3af2efc64a37ac98f8366d20e0e8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41141 } } }
I20260924 15:27:57.962980 10923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.963070 10925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:57.963315 10932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:57.964108 10932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:57.965780 10932 catalog_manager.cc:1347] Generated new cluster ID: d228633d2da3484281fe4a4fede4d7f7
I20260924 15:27:57.965834 10932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:57.971978 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.959989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.5.98.254:41141)
0924 15:27:57.960258 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.960268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.960303 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:27:57.960422 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.960429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.960509 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.960664 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:57.960939 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.962743 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.963880 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.964485 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.964608 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:57.965343 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:57.965347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:57.965347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:57.965544 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:57.965584 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:57.965609 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:57.965683 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:57.971756 (+ 6073us) server_negotiation.cc:299] Negotiation successful
0924 15:27:57.971831 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260924 15:27:57.972942 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:57.982815 10932 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:57.983317 10932 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:57.986999 10932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 786a3af2efc64a37ac98f8366d20e0e8: Generated new TSK 0
I20260924 15:27:57.987172 10932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:58.001714 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:57.997969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.5.98.254:41141)
0924 15:27:57.998029 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:57.998031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:57.998109 (+ 78us) server_negotiation.cc:407] Connection header received
0924 15:27:57.998190 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:57.998191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:57.998203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:57.998272 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:27:57.998392 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.999167 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.000016 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.000322 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.000387 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:58.000560 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:27:58.001500 (+ 940us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:27:58.001644 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48548: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 15:27:58.001775 10922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48548: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 15:27:58.003724 10947 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:57.997861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41141 (local address 127.0.0.1:48548)
0924 15:27:57.998050 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:57.998060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:57.998121 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:57.998281 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:57.998281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:57.998283 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:27:57.998330 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.998332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:57.999257 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:57.999258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:57.999940 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:57.999942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.000406 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.000407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:58.000448 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:58.000452 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:27:58.001609 (+ 1157us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:27:58.003593 (+ 1984us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.98.254:41141: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1772,"thread_start_us":121,"threads_started":1}
I20260924 15:27:58.004134 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10851
E20260924 15:27:58.014611 10948 webserver.cc:533] Webserver: sslize failed:
E20260924 15:27:58.014690 10948 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T15:27:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T15:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:58Z Disabled control of system clock
I20260924 15:27:58.050238 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:39897
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:39897
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/testchainca.cert with env {}
W20260924 15:27:58.129498 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:58.129705 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:58.129727 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:58.130934 10952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:58.130977 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:58.130990 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:58.131003 10952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:58.131026 10952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:27:58.131038 10952 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:27:58.131085 10952 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 15:27:58.132774 10952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:39897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:58.133034 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:58.133262 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:58.138141 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:58.138269 10952 server_base.cc:1034] running on GCE node
W20260924 15:27:58.138180 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:58.138149 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:58.138610 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:58.138855 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:58.140021 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790263678139997 us; error 55 us; skew 500 ppm
I20260924 15:27:58.141214 10952 webserver.cc:460] Webserver started at http://127.5.98.254:36695/ using document root <none> and password file <none>
I20260924 15:27:58.141459 10952 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:58.141532 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:58.141683 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:58.142715 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "8e76e79e03eb4a17977d02ce3ca93b9b"
format_stamp: "Formatted at 2026-09-24 15:27:58 on dist-test-slave-4g6f"
server_key: "ead8997db9492d8cb6e76aca68e515a1"
server_key_iv: "a782869b5d342f896295ae6d5ae2d71b"
server_key_version: "encryptionkey@0"
I20260924 15:27:58.143078 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "8e76e79e03eb4a17977d02ce3ca93b9b"
format_stamp: "Formatted at 2026-09-24 15:27:58 on dist-test-slave-4g6f"
server_key: "ead8997db9492d8cb6e76aca68e515a1"
server_key_iv: "a782869b5d342f896295ae6d5ae2d71b"
server_key_version: "encryptionkey@0"
I20260924 15:27:58.144527 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:58.145251 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:58.145463 10952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:58.145561 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "8e76e79e03eb4a17977d02ce3ca93b9b"
format_stamp: "Formatted at 2026-09-24 15:27:58 on dist-test-slave-4g6f"
server_key: "ead8997db9492d8cb6e76aca68e515a1"
server_key_iv: "a782869b5d342f896295ae6d5ae2d71b"
server_key_version: "encryptionkey@0"
I20260924 15:27:58.145648 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:58.171759 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:58.172083 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:58.172242 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:58.176481 10952 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:39897
I20260924 15:27:58.176527 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:39897 every 8 connection(s)
I20260924 15:27:58.176895 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:58.177839 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:58.180187 11020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b: Bootstrap starting.
I20260924 15:27:58.180765 11020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:58.181049 11020 log.cc:824] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:58.181716 11020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b: No bootstrap required, opened a new log
I20260924 15:27:58.183264 11020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } }
I20260924 15:27:58.183454 11020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:58.183519 11020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76e79e03eb4a17977d02ce3ca93b9b, State: Initialized, Role: FOLLOWER
I20260924 15:27:58.183732 11020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } }
I20260924 15:27:58.183815 11020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:58.183859 11020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:58.183910 11020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:58.184437 11020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } }
I20260924 15:27:58.184561 11020 leader_election.cc:302] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e76e79e03eb4a17977d02ce3ca93b9b; no voters:
I20260924 15:27:58.184674 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 10952
I20260924 15:27:58.184768 11020 leader_election.cc:288] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:58.184810 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:58.184793 11023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:58.184947 11023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [term 1 LEADER]: Becoming Leader. State: Replica: 8e76e79e03eb4a17977d02ce3ca93b9b, State: Running, Role: LEADER
I20260924 15:27:58.185009 5515 external_mini_cluster.cc:1428] Setting key c0f2b357936307a69ccd40e042cf3f8b
I20260924 15:27:58.185032 11023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } }
I20260924 15:27:58.185087 11020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:58.185344 11024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e76e79e03eb4a17977d02ce3ca93b9b. Latest consensus state: current_term: 1 leader_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } } }
I20260924 15:27:58.185431 11024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:58.185344 11025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e79e03eb4a17977d02ce3ca93b9b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 39897 } } }
I20260924 15:27:58.185475 11025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:58.185834 11029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:58.186388 11029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:58.191327 11029 catalog_manager.cc:1347] Generated new cluster ID: 4441c497c053434aa18f54ddf7eb5ccf
I20260924 15:27:58.191334 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:58.185833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.5.98.254:39897)
0924 15:27:58.186174 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:58.186184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:58.186214 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:27:58.186331 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:58.186338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:58.186412 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:58.186822 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:58.186926 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.188020 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.189135 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.189534 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.189619 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:58.190371 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:58.190375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:58.190375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:58.190576 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:58.190631 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:58.190649 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:58.190764 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:58.191021 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 15:27:58.191109 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":68,"threads_started":1}
I20260924 15:27:58.191426 11029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:58.192533 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:58.198256 11029 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:58.198901 11029 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:58.207304 11029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e76e79e03eb4a17977d02ce3ca93b9b: Generated new TSK 0
I20260924 15:27:58.207481 11029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:58.263567 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:58.219249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.5.98.254:39897)
0924 15:27:58.219300 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:58.219304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:58.219362 (+ 58us) server_negotiation.cc:407] Connection header received
0924 15:27:58.219450 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:58.219451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:58.219461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:58.219520 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:27:58.219699 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.220187 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.221312 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.221502 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.221549 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:58.221737 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:27:58.263311 (+ 41574us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:27:58.263507 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 15:27:58.263693 11034 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 15:27:58.263763 11049 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:58.219133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:39897 (local address 127.0.0.1:59044)
0924 15:27:58.219303 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:58.219314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:58.219381 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:58.219533 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:58.219534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:58.219537 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:27:58.219599 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:58.219600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.220264 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.220265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:58.221229 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:58.221232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:58.221578 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:58.221579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:58.221641 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:58.221646 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:27:58.263469 (+ 41823us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:27:58.263643 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.98.254:39897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260924 15:27:58.264140 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 10952
2026-09-24T15:27:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (238 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:58 dist-test-slave-4g6f krb5kdc[11061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:58 dist-test-slave-4g6f krb5kdc[11061](info): set up 2 sockets
Sep 24 15:27:58 dist-test-slave-4g6f krb5kdc[11061](info): commencing operation
krb5kdc: starting...
W20260924 15:27:59.478744 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:27:59 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:27:59.598568 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42781
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:59.678699 11074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:59.678933 11074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:59.678977 11074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:59.680193 11074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:59.680254 11074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:59.680279 11074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:59.680299 11074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:59.680328 11074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:59.681775 11074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:59.682072 11074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:59.682299 11074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:59.687345 11080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:59.687433 11079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:59.687361 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:59.687487 11074 server_base.cc:1034] running on GCE node
I20260924 15:27:59.687881 11074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:59.688176 11074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:59.689494 11074 hybrid_clock.cc:648] HybridClock initialized: now 1790263679689482 us; error 43 us; skew 500 ppm
Sep 24 15:27:59 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:59.692018 11074 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:27:59.692531 11074 webserver.cc:460] Webserver started at http://127.5.98.254:43715/ using document root <none> and password file <none>
I20260924 15:27:59.692791 11074 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:59.692866 11074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:59.693028 11074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:59.694183 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "0904a5c82d964d998f60caa802a58567"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "701183729681cdd14f45e5597c9a9114"
server_key_iv: "3226cb1b1acb84206c55ebe4bcca7c81"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.694595 11074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "0904a5c82d964d998f60caa802a58567"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "701183729681cdd14f45e5597c9a9114"
server_key_iv: "3226cb1b1acb84206c55ebe4bcca7c81"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.696214 11074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:59.697012 11089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:59.697181 11074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:59.697286 11074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "0904a5c82d964d998f60caa802a58567"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "701183729681cdd14f45e5597c9a9114"
server_key_iv: "3226cb1b1acb84206c55ebe4bcca7c81"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.697366 11074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:59.714603 11074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:59.716590 11074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:59.716751 11074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:59.720700 11074 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:42781
I20260924 15:27:59.720736 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:42781 every 8 connection(s)
I20260924 15:27:59.721091 11074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:59.721896 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:59.722720 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11074
I20260924 15:27:59.722805 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:27:59.722990 5515 external_mini_cluster.cc:1428] Setting key 5a3ba958bcabe7fb656fcf7356b0bb3e
I20260924 15:27:59.725144 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567: Bootstrap starting.
I20260924 15:27:59.725855 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:59.726147 11142 log.cc:824] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:59.726838 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567: No bootstrap required, opened a new log
I20260924 15:27:59.729097 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } }
I20260924 15:27:59.729264 11142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:59.729301 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0904a5c82d964d998f60caa802a58567, State: Initialized, Role: FOLLOWER
I20260924 15:27:59.729465 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } }
I20260924 15:27:59.729549 11142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:59.729583 11142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:59.729637 11142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:27:59 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:59.730329 11142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } }
I20260924 15:27:59.730449 11142 leader_election.cc:302] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0904a5c82d964d998f60caa802a58567; no voters:
I20260924 15:27:59.730646 11142 leader_election.cc:288] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:59.730693 11147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:59.730839 11147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [term 1 LEADER]: Becoming Leader. State: Replica: 0904a5c82d964d998f60caa802a58567, State: Running, Role: LEADER
I20260924 15:27:59.730906 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:59.730954 11147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } }
I20260924 15:27:59.731256 11147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0904a5c82d964d998f60caa802a58567. Latest consensus state: current_term: 1 leader_uuid: "0904a5c82d964d998f60caa802a58567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } } }
I20260924 15:27:59.731338 11147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:59.731372 11149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0904a5c82d964d998f60caa802a58567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0904a5c82d964d998f60caa802a58567" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42781 } } }
I20260924 15:27:59.731429 11149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:59.731603 11153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:59.733222 11153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:59.734926 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:59.723523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.5.98.254:42781)
0924 15:27:59.723734 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:59.723744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:59.723777 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:27:59.724362 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:59.724372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:59.724439 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:59.724612 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:59.725156 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.726504 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.727276 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.727897 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.728011 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:59.730477 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:59.730488 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:59.730488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:59.730515 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:59.732718 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:59.733148 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:59.733152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:59.733152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:59.733280 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:59.733518 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:59.733519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:59.733519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:59.733835 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:59.733987 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:59.734513 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 15:27:59.734658 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260924 15:27:59.736044 11153 catalog_manager.cc:1347] Generated new cluster ID: 2bff9040963c4c82b24a291f6d4972fd
I20260924 15:27:59.736104 11153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:59.742255 11153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:59.742735 11153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:59.747995 11153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0904a5c82d964d998f60caa802a58567: Generated new TSK 0
I20260924 15:27:59.748193 11153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:27:59.774945 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:59.857683 11168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:59.857889 11168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:59.857915 11168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:59.859134 11168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:59.859177 11168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:59.859228 11168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:27:59.860742 11168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:59.860998 11168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:59.861239 11168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:59.866289 11174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:59.866362 11176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:59.866400 11168 server_base.cc:1034] running on GCE node
W20260924 15:27:59.866362 11173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:59.866811 11168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:59.867020 11168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:59.868166 11168 hybrid_clock.cc:648] HybridClock initialized: now 1790263679868147 us; error 41 us; skew 500 ppm
Sep 24 15:27:59 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:59.870694 11168 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:27:59.871100 11168 webserver.cc:460] Webserver started at http://127.5.98.193:45137/ using document root <none> and password file <none>
I20260924 15:27:59.871270 11168 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:59.871308 11168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:59.871394 11168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:59.872412 11168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "177a0e171d374192acf7bee729b05a1f"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "50503864375d2aa6af68c3ecc05b9e10"
server_key_iv: "33a00115b65d445b9797e6654073db7c"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.872706 11168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "177a0e171d374192acf7bee729b05a1f"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "50503864375d2aa6af68c3ecc05b9e10"
server_key_iv: "33a00115b65d445b9797e6654073db7c"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.873932 11168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:59.874616 11183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:59.874789 11168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:59.874893 11168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "177a0e171d374192acf7bee729b05a1f"
format_stamp: "Formatted at 2026-09-24 15:27:59 on dist-test-slave-4g6f"
server_key: "50503864375d2aa6af68c3ecc05b9e10"
server_key_iv: "33a00115b65d445b9797e6654073db7c"
server_key_version: "encryptionkey@0"
I20260924 15:27:59.874969 11168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:59.886891 11168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:59.889156 11168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:59.889309 11168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:59.889516 11168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:59.889818 11168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:59.889871 11168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:59.889911 11168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:59.889940 11168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:59.895771 11168 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:38901
I20260924 15:27:59.895807 11297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:38901 every 8 connection(s)
I20260924 15:27:59.896143 11168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:59.899576 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11168
I20260924 15:27:59.899706 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:59.899883 5515 external_mini_cluster.cc:1428] Setting key 7a7a124e1d77008c8542e9c6ea71b43a
Sep 24 15:27:59 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:27:59.905486 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:59.896879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:42191 (local address 127.5.98.254:42781)
0924 15:27:59.896957 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:59.896960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:59.897187 (+ 227us) server_negotiation.cc:407] Connection header received
0924 15:27:59.898104 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:59.898106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:59.898116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:59.898166 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:59.898714 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.899207 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.900355 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.900577 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.900665 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:59.902713 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:59.902714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:59.902714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:59.902722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:59.904145 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:59.904565 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:59.904566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:59.904567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:59.904624 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:59.904870 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:59.904871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:59.904871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:59.904979 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:59.905045 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:59.905315 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 15:27:59.905398 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 15:27:59.905674 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:59.896857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42781 (local address 127.5.98.193:42191)
0924 15:27:59.897024 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:59.897058 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:59.898012 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:59.898218 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:59.898221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:59.898445 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:59.898634 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:59.898639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.899283 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.899284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:59.900264 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:59.900267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:59.900676 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:59.900677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:59.900807 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:59.901308 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:59.901316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:59.902607 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:59.904260 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:59.904267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:59.904270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:59.904479 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:59.904695 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:59.904695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:59.904696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:59.904798 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:59.905049 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:59.905056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:59.905183 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 15:27:59.905294 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:27:59.905436 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260924 15:27:59.906172 11298 heartbeater.cc:347] Connected to a master server at 127.5.98.254:42781
I20260924 15:27:59.906260 11298 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:59.906448 11298 heartbeater.cc:511] Master 127.5.98.254:42781 requested a full tablet report, sending...
I20260924 15:27:59.907024 11106 ts_manager.cc:194] Registered new tserver with Master: 177a0e171d374192acf7bee729b05a1f (127.5.98.193:38901)
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
I20260924 15:27:59.908010 11106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:42191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:27:59.926084 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:00.014868 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:00.015170 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:00.015210 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:00.016453 11303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:00.016517 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:00.016563 11303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:00.018064 11303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:00.018329 11303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:00.018568 11303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:00.023794 11309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:00.023794 11308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:00.024013 11303 server_base.cc:1034] running on GCE node
W20260924 15:28:00.023797 11311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:00.024328 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:00.024590 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:00.025758 11303 hybrid_clock.cc:648] HybridClock initialized: now 1790263680025753 us; error 35 us; skew 500 ppm
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.028513 11303 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:28:00.028965 11303 webserver.cc:460] Webserver started at http://127.5.98.194:37181/ using document root <none> and password file <none>
I20260924 15:28:00.029219 11303 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:00.029294 11303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:00.029470 11303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:00.030639 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "b7428741ff1f4fbdb8777c1f7ff264f8"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "911687d0ca345c7c18e534313b34051d"
server_key_iv: "e7993a1af6bad72ad942fafb073ac08f"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.031036 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "b7428741ff1f4fbdb8777c1f7ff264f8"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "911687d0ca345c7c18e534313b34051d"
server_key_iv: "e7993a1af6bad72ad942fafb073ac08f"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.032608 11303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:00.033389 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.033567 11303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:00.033663 11303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "b7428741ff1f4fbdb8777c1f7ff264f8"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "911687d0ca345c7c18e534313b34051d"
server_key_iv: "e7993a1af6bad72ad942fafb073ac08f"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.033748 11303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:00.043442 11303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:00.045853 11303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:00.046046 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:00.046293 11303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:00.046612 11303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:00.046689 11303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.046742 11303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:00.046772 11303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.053171 11303 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:43783
I20260924 15:28:00.053223 11432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:43783 every 8 connection(s)
I20260924 15:28:00.053556 11303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:00.060724 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11303
I20260924 15:28:00.060850 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:00.061074 5515 external_mini_cluster.cc:1428] Setting key bb3cadfae01e765632cf1e1b111e2f37
I20260924 15:28:00.064043 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.054307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:53069 (local address 127.5.98.254:42781)
0924 15:28:00.054353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.054355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.054613 (+ 258us) server_negotiation.cc:407] Connection header received
0924 15:28:00.055642 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.055644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.055731 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.055807 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.056532 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.057066 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.058134 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.058335 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.058429 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.060733 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:00.060735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:00.060735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:00.060750 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:00.062371 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.062869 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.062871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.062871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.062935 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.063291 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.063292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.063293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.063449 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:00.063522 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:00.063838 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 15:28:00.063932 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:28:00.064126 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.054297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42781 (local address 127.5.98.194:53069)
0924 15:28:00.054489 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:00.054524 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:00.055516 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:00.055904 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:00.055908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:00.056163 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:00.056412 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:00.056417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.057175 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.057176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:00.058018 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:00.058021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.058449 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.058450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:00.058578 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.059110 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:00.059119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:00.060619 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:00.062481 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:00.062487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:00.062490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:00.062779 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:00.063047 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:00.063048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:00.063048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:00.063208 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:00.063521 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:00.063526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:00.063647 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 15:28:00.063801 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 15:28:00.063934 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260924 15:28:00.064626 11433 heartbeater.cc:347] Connected to a master server at 127.5.98.254:42781
I20260924 15:28:00.064731 11433 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:00.064939 11433 heartbeater.cc:511] Master 127.5.98.254:42781 requested a full tablet report, sending...
I20260924 15:28:00.065335 11106 ts_manager.cc:194] Registered new tserver with Master: b7428741ff1f4fbdb8777c1f7ff264f8 (127.5.98.194:43783)
I20260924 15:28:00.065793 11106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:53069
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:28:00.087146 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:00.166965 11438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:00.167150 11438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:00.167172 11438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:00.168321 11438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:00.168360 11438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:00.168386 11438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:00.169824 11438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:42781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:00.170077 11438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:00.170306 11438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:00.175246 11446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:00.175251 11444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:00.175268 11443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:00.175453 11438 server_base.cc:1034] running on GCE node
I20260924 15:28:00.175778 11438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:00.176025 11438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:00.177192 11438 hybrid_clock.cc:648] HybridClock initialized: now 1790263680177178 us; error 39 us; skew 500 ppm
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.179895 11438 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:28:00.180331 11438 webserver.cc:460] Webserver started at http://127.5.98.195:39117/ using document root <none> and password file <none>
I20260924 15:28:00.180604 11438 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:00.180675 11438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:00.180821 11438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:00.181895 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "6d4c1047ecb6440f84ca64c564779560"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "63c93bc75baa12ddf524f606cdcfa651"
server_key_iv: "32d6afd227248e736f7c2f7876c0df2a"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.182264 11438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4c1047ecb6440f84ca64c564779560"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "63c93bc75baa12ddf524f606cdcfa651"
server_key_iv: "32d6afd227248e736f7c2f7876c0df2a"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.183605 11438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:28:00.184641 11453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.184811 11438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:00.184911 11438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "6d4c1047ecb6440f84ca64c564779560"
format_stamp: "Formatted at 2026-09-24 15:28:00 on dist-test-slave-4g6f"
server_key: "63c93bc75baa12ddf524f606cdcfa651"
server_key_iv: "32d6afd227248e736f7c2f7876c0df2a"
server_key_version: "encryptionkey@0"
I20260924 15:28:00.184993 11438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:00.204221 11438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:00.205938 11438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:00.206101 11438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:00.206329 11438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:00.206638 11438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:00.206691 11438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.206725 11438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:00.206750 11438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:00.212455 11438 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46073
I20260924 15:28:00.212505 11566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46073 every 8 connection(s)
I20260924 15:28:00.212797 11438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:00.221745 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.213512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:53471 (local address 127.5.98.254:42781)
0924 15:28:00.213557 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.213560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.213750 (+ 190us) server_negotiation.cc:407] Connection header received
0924 15:28:00.214650 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.214651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.214660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.214709 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.215285 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.215849 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.216823 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.217017 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.217065 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.219115 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:00.219116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:00.219116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:00.219123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:00.220588 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.220989 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.220990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.220990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.221038 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.221253 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.221254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.221254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.221351 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:00.221401 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:00.221624 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 15:28:00.221680 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:28:00.222033 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11438
I20260924 15:28:00.222131 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:00.222304 5515 external_mini_cluster.cc:1428] Setting key 49e311ed718038f7df0edc2ce7e58c7b
I20260924 15:28:00.222606 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.213462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42781 (local address 127.5.98.195:53471)
0924 15:28:00.213619 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:00.213659 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:00.214546 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:00.214766 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:00.214769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:00.214997 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:00.215211 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:00.215215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.215928 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.215929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:00.216747 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:00.216749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.217089 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.217090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:00.217192 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.217599 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:00.217607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:00.219023 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:00.220679 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:00.220686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:00.220688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:00.220902 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:00.221097 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:00.221098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:00.221098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:00.221196 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:00.221405 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:00.221409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:00.221518 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:28:00.221610 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 15:28:00.221726 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260924 15:28:00.223029 11567 heartbeater.cc:347] Connected to a master server at 127.5.98.254:42781
I20260924 15:28:00.223130 11567 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:00.223312 11567 heartbeater.cc:511] Master 127.5.98.254:42781 requested a full tablet report, sending...
I20260924 15:28:00.223749 11106 ts_manager.cc:194] Registered new tserver with Master: 6d4c1047ecb6440f84ca64c564779560 (127.5.98.195:46073)
I20260924 15:28:00.224090 11106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:53471
I20260924 15:28:00.234163 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/test-admin.keytab.
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.249974 5515 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 15:28:00.251765 11574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:00.250854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.5.98.193:38901)
0924 15:28:00.251025 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.251030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.251040 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:00.251066 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.251068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.251082 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.251148 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.251624 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54184: BlockingRecv error: recv got EOF from 127.0.0.1:54184 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
W20260924 15:28:00.251870 5515 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.252854 5515 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.253748 5515 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:28:00.262034 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.254281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.5.98.193:38901)
0924 15:28:00.254341 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.254343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.254459 (+ 116us) server_negotiation.cc:407] Connection header received
0924 15:28:00.254482 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.254483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.254501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.254562 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.254973 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.255795 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.256544 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.256891 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.256984 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.259062 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:00.259070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:00.259070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:00.259078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:00.260772 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.261150 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.261153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.261154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.261230 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.261477 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.261478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.261478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.261614 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:00.261760 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:00.261841 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:28:00.261923 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:28:00.263996 11574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:00.263257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.5.98.193:38901)
0924 15:28:00.263316 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.263318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.263364 (+ 46us) server_negotiation.cc:407] Connection header received
0924 15:28:00.263487 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.263488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.263499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.263551 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.263927 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54200: BlockingRecv error: recv got EOF from 127.0.0.1:54200 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 15:28:00.264099 5515 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.265089 5515 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.265955 5515 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:28:00.272714 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.266421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.5.98.193:38901)
0924 15:28:00.266476 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.266478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.266562 (+ 84us) server_negotiation.cc:407] Connection header received
0924 15:28:00.266595 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.266596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.266603 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.266647 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.266956 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.267455 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.268125 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.268314 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.268364 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.270259 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:00.270261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:00.270261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:00.270267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:00.271764 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.272073 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.272073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.272074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.272121 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.272280 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.272280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.272280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.272361 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:00.272450 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:00.272513 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 15:28:00.272567 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 15:28:00.274174 11574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:00.273699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.5.98.193:38901)
0924 15:28:00.273754 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.273756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.273778 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:28:00.273826 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.273827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.273833 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.273875 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.274121 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54218: BlockingRecv error: recv got EOF from 127.0.0.1:54218 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:28:00.274233 5515 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263655576751-5515-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.275097 5515 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:00.275949 5515 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:28:00.282380 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:00.276424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.5.98.193:38901)
0924 15:28:00.276491 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:00.276493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:00.276500 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:00.276536 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:00.276537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:00.276544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:00.276588 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:00.276886 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.277384 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.278069 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:00.278259 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:00.278309 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:00.280148 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:00.280149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:00.280149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:00.280155 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:00.281475 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.281764 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.281765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.281765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.281811 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:00.281960 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:00.281960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:00.281960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:00.282032 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:00.282090 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:00.282206 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:28:00.282253 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:28:00.282981 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11168
I20260924 15:28:00.288422 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11303
I20260924 15:28:00.292937 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11438
I20260924 15:28:00.297216 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11074
2026-09-24T15:28:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1859 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11587](info): set up 2 sockets
Sep 24 15:28:00 dist-test-slave-4g6f krb5kdc[11587](info): commencing operation
krb5kdc: starting...
W20260924 15:28:01.336812 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab.
Entry for principal oryx/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab.
I20260924 15:28:01.455116 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41153
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab
--principal=oryx/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:01.535579 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:01.535802 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:01.535827 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:01.536890 11600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:01.536924 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:01.536944 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:01.536957 11600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:01.536969 11600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:01.538328 11600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab
--principal=oryx/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:01.538512 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:01.538678 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:01.543610 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:01.543598 11608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:01.543601 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:01.543951 11600 server_base.cc:1034] running on GCE node
I20260924 15:28:01.544190 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:01.544464 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:01.545615 11600 hybrid_clock.cc:648] HybridClock initialized: now 1790263681545599 us; error 31 us; skew 500 ppm
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:01.548190 11600 init.cc:377] Logged in from keytab as oryx/127.5.98.254@KRBTEST.COM (short username oryx)
I20260924 15:28:01.548697 11600 webserver.cc:460] Webserver started at http://127.5.98.254:37107/ using document root <none> and password file <none>
I20260924 15:28:01.548934 11600 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:01.549012 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:01.549180 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:01.550303 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "51cd0b5a18634343ab87689c36f3b53b"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "f5de2caf9f2fed58ad655adfdaa9a871"
server_key_iv: "045103d5300f6e07d80e237305dd79b5"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.550697 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "51cd0b5a18634343ab87689c36f3b53b"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "f5de2caf9f2fed58ad655adfdaa9a871"
server_key_iv: "045103d5300f6e07d80e237305dd79b5"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.552224 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:01.552975 11615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.553153 11600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:01.553256 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "51cd0b5a18634343ab87689c36f3b53b"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "f5de2caf9f2fed58ad655adfdaa9a871"
server_key_iv: "045103d5300f6e07d80e237305dd79b5"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.553349 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:01.586903 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:01.588924 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:01.589113 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:01.593410 11600 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:41153
I20260924 15:28:01.593446 11667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:41153 every 8 connection(s)
I20260924 15:28:01.593842 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:01.594825 11668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:01.597151 11668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Bootstrap starting.
I20260924 15:28:01.597733 11668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:01.598009 11668 log.cc:824] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:01.598707 11668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: No bootstrap required, opened a new log
I20260924 15:28:01.599843 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11600
I20260924 15:28:01.599963 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:01.600155 5515 external_mini_cluster.cc:1428] Setting key dff40685b505c772874f70f5f083825b
I20260924 15:28:01.600641 11668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } }
I20260924 15:28:01.600879 11668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:01.600941 11668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cd0b5a18634343ab87689c36f3b53b, State: Initialized, Role: FOLLOWER
I20260924 15:28:01.601064 11668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } }
I20260924 15:28:01.601140 11668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:01.601161 11668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:01.601222 11668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:01.602049 11668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } }
I20260924 15:28:01.602144 11668 leader_election.cc:302] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cd0b5a18634343ab87689c36f3b53b; no voters:
I20260924 15:28:01.602334 11668 leader_election.cc:288] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:01.602362 11673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:01.602465 11673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 LEADER]: Becoming Leader. State: Replica: 51cd0b5a18634343ab87689c36f3b53b, State: Running, Role: LEADER
I20260924 15:28:01.602545 11673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } }
I20260924 15:28:01.602598 11668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:01.602855 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } }
I20260924 15:28:01.603009 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:01.603216 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cd0b5a18634343ab87689c36f3b53b. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } }
I20260924 15:28:01.603364 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:01.603601 11680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:01.604249 11680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:01.605984 11680 catalog_manager.cc:1347] Generated new cluster ID: 661240a6ed4a431e8088fd83cee4e97a
I20260924 15:28:01.606078 11680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:01.610512 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:01.600690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.5.98.254:41153)
0924 15:28:01.600879 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:01.600888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:01.600924 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:28:01.601065 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:01.601073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:01.601154 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:01.601290 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:01.601948 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.603257 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.604370 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.604963 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.605067 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:01.607415 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:01.607419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:01.607419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:01.607431 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:01.609059 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.609432 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.609436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.609436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.609531 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.609741 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.609742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.609743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.609972 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:01.610158 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:01.610225 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:28:01.610362 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260924 15:28:01.619980 11680 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:01.620652 11680 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:01.631592 11680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Generated new TSK 0
I20260924 15:28:01.631791 11680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
I20260924 15:28:01.666980 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:01.750422 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:01.750622 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:01.750645 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:01.751871 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:01.751916 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:01.751967 11694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:01.753671 11694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:01.753857 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:01.754050 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:01.758992 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:01.759040 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:01.759052 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:01.759194 11694 server_base.cc:1034] running on GCE node
I20260924 15:28:01.759433 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:01.759714 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:01.760895 11694 hybrid_clock.cc:648] HybridClock initialized: now 1790263681760885 us; error 34 us; skew 500 ppm
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:01.763447 11694 init.cc:377] Logged in from keytab as oryx/127.5.98.193@KRBTEST.COM (short username oryx)
I20260924 15:28:01.763922 11694 webserver.cc:460] Webserver started at http://127.5.98.193:41761/ using document root <none> and password file <none>
I20260924 15:28:01.764142 11694 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:01.764204 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:01.764354 11694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:01.765453 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "f52f867273704793b3a3b4aca920afcf"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "1deb24b15b26fe2d6c20fdcf3e375d50"
server_key_iv: "cc302ac49f53b600deb4eeffe514cfc2"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.765817 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "f52f867273704793b3a3b4aca920afcf"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "1deb24b15b26fe2d6c20fdcf3e375d50"
server_key_iv: "cc302ac49f53b600deb4eeffe514cfc2"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.767128 11694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:28:01.767964 11709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.768142 11694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:01.768239 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "f52f867273704793b3a3b4aca920afcf"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "1deb24b15b26fe2d6c20fdcf3e375d50"
server_key_iv: "cc302ac49f53b600deb4eeffe514cfc2"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.768318 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:01.776793 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:01.778510 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:01.778663 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:01.778882 11694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:01.779193 11694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:01.779244 11694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.779277 11694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:01.779304 11694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.785101 11694 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:45047
I20260924 15:28:01.785145 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:45047 every 8 connection(s)
I20260924 15:28:01.785533 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:01.791695 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11694
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:01.791805 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:01.792004 5515 external_mini_cluster.cc:1428] Setting key 37c10e9b710cd407460ad7e5141d777a
I20260924 15:28:01.795182 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:01.786233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:33591 (local address 127.5.98.254:41153)
0924 15:28:01.786290 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:01.786292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:01.786479 (+ 187us) server_negotiation.cc:407] Connection header received
0924 15:28:01.787105 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:01.787106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:01.787122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:01.787181 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:01.788196 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.788742 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.789757 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.789966 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.790038 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:01.792324 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:01.792326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:01.792326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:01.792336 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:01.793775 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.794196 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.794197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.794197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.794259 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.794517 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.794518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.794519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.794641 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:01.794713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:01.795000 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 15:28:01.795082 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:28:01.795384 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:01.786218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:33591)
0924 15:28:01.786368 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:01.786402 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:01.786899 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:01.787259 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:01.787262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:01.787907 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:01.788095 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:01.788100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.788838 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.788839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:01.789660 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:01.789662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.790044 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.790045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:01.790167 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:01.790673 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:01.790682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:01.792216 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:01.793879 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:01.793886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:01.793889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:01.794119 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:01.794326 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:01.794327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:01.794327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:01.794444 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:01.794719 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:01.794726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:01.794860 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 15:28:01.794965 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:28:01.795138 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 15:28:01.795887 11823 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:01.796005 11823 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:01.796216 11823 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:01.796809 11632 ts_manager.cc:194] Registered new tserver with Master: f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
I20260924 15:28:01.797801 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:33591
WARNING: no policy specified for oryx/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
I20260924 15:28:01.818138 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:01.898859 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:01.899057 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:01.899082 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:01.900159 11828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:01.900192 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:01.900218 11828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:01.901551 11828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:01.901722 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:01.901911 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:01.907071 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:01.907059 11833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:01.907171 11828 server_base.cc:1034] running on GCE node
W20260924 15:28:01.907123 11834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:01.907478 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:01.907759 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:01.908922 11828 hybrid_clock.cc:648] HybridClock initialized: now 1790263681908918 us; error 36 us; skew 500 ppm
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:01.911324 11828 init.cc:377] Logged in from keytab as oryx/127.5.98.194@KRBTEST.COM (short username oryx)
I20260924 15:28:01.911814 11828 webserver.cc:460] Webserver started at http://127.5.98.194:33047/ using document root <none> and password file <none>
I20260924 15:28:01.911998 11828 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:01.912057 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:01.912185 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:01.913154 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "783574c66bc04c82a98647ad9e156ba6"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "b5808dfb6b898b192460e280d4757eae"
server_key_iv: "80057fff9542baa7964d99c49d1c0232"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.913448 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "783574c66bc04c82a98647ad9e156ba6"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "b5808dfb6b898b192460e280d4757eae"
server_key_iv: "80057fff9542baa7964d99c49d1c0232"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.914701 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:01.915396 11843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.915555 11828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:01.915637 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "783574c66bc04c82a98647ad9e156ba6"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "b5808dfb6b898b192460e280d4757eae"
server_key_iv: "80057fff9542baa7964d99c49d1c0232"
server_key_version: "encryptionkey@0"
I20260924 15:28:01.915738 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:01.927747 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:01.929940 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:01.930059 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:01.930243 11828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:01.930496 11828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:01.930526 11828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.930547 11828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:01.930559 11828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:01.935999 11828 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:37111
I20260924 15:28:01.936056 11956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:37111 every 8 connection(s)
I20260924 15:28:01.936383 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:28:01 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:01.942610 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11828
I20260924 15:28:01.942716 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:01.942930 5515 external_mini_cluster.cc:1428] Setting key 9faaa7d141a3a1330e4ac8aafe5f5484
I20260924 15:28:01.945520 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:01.937116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:38405 (local address 127.5.98.254:41153)
0924 15:28:01.937179 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:01.937181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:01.937354 (+ 173us) server_negotiation.cc:407] Connection header received
0924 15:28:01.937811 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:01.937812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:01.937822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:01.937881 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:01.938780 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.939266 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.940301 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.940516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.940573 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:01.942483 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:01.942484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:01.942485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:01.942490 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:01.944183 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.944562 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.944563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.944564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.944629 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:01.944845 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:01.944846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:01.944847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:01.944991 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:01.945063 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:01.945394 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 15:28:01.945459 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:28:01.945792 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:01.937064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:38405)
0924 15:28:01.937227 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:01.937267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:01.937732 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:01.937950 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:01.937955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:01.938516 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:01.938705 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:01.938709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.939340 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.939341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:01.940227 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:01.940230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:01.940587 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:01.940588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:01.940687 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:01.941102 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:01.941110 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:01.942380 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:01.944258 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:01.944263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:01.944265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:01.944491 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:01.944694 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:01.944694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:01.944695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:01.944787 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:01.945063 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:01.945069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:01.945242 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 15:28:01.945364 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 15:28:01.945535 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260924 15:28:01.946319 11957 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:01.946429 11957 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:01.946604 11957 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:01.946983 11632 ts_manager.cc:194] Registered new tserver with Master: 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111)
I20260924 15:28:01.947355 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:38405
WARNING: no policy specified for oryx/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
I20260924 15:28:01.966858 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:02.047874 11962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:02.048056 11962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:02.048079 11962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:02.049203 11962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:02.049243 11962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:02.049289 11962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:02.050678 11962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:02.050896 11962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:02.051108 11962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.055932 11967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.055943 11970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.056079 11962 server_base.cc:1034] running on GCE node
W20260924 15:28:02.056196 11968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.056380 11962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.056623 11962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.057778 11962 hybrid_clock.cc:648] HybridClock initialized: now 1790263682057756 us; error 49 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.060199 11962 init.cc:377] Logged in from keytab as oryx/127.5.98.195@KRBTEST.COM (short username oryx)
I20260924 15:28:02.060647 11962 webserver.cc:460] Webserver started at http://127.5.98.195:35435/ using document root <none> and password file <none>
I20260924 15:28:02.060884 11962 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.060948 11962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.061079 11962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:02.062088 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "9269a6f59fe94cfc99757bc2276b1a2d"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "ae3b4aa3c1d74d6e2aa90e8d7efd499e"
server_key_iv: "71e241384e2a181cb3bfd8d2c86b11c4"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.062450 11962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "9269a6f59fe94cfc99757bc2276b1a2d"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "ae3b4aa3c1d74d6e2aa90e8d7efd499e"
server_key_iv: "71e241384e2a181cb3bfd8d2c86b11c4"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.063854 11962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:28:02.064669 11977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.064863 11962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:02.064950 11962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "9269a6f59fe94cfc99757bc2276b1a2d"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "ae3b4aa3c1d74d6e2aa90e8d7efd499e"
server_key_iv: "71e241384e2a181cb3bfd8d2c86b11c4"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.065037 11962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.089084 11962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.091367 11962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:02.091528 11962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.091826 11962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:02.092151 11962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:02.092201 11962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.092237 11962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:02.092264 11962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.098156 11962 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:39463
I20260924 15:28:02.098217 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:39463 every 8 connection(s)
I20260924 15:28:02.098546 11962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:02.101431 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 11962
I20260924 15:28:02.101511 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:02.101687 5515 external_mini_cluster.cc:1428] Setting key 84116089ebfd6744008324a754d763b4
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:02.108124 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.099338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:54987 (local address 127.5.98.254:41153)
0924 15:28:02.099413 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.099416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.099585 (+ 169us) server_negotiation.cc:407] Connection header received
0924 15:28:02.100071 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.100072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.100082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.100138 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.101059 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.101597 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.102753 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.102969 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.103039 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.105319 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.105321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.105321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.105329 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.106735 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.107154 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.107155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.107155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.107212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.107467 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.107468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.107468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.107582 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.107662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.107950 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.108034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:28:02.108326 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.099295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:54987)
0924 15:28:02.099470 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.099506 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.099987 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.100191 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.100195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.100788 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.100984 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.100990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.101699 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.101700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.102639 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.102642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.103058 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.103059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.103214 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.103812 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.103823 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.105225 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.106833 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.106840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.106842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.107065 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.107278 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.107279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.107279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.107399 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.107674 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.107681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.107819 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.107928 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.108083 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 15:28:02.108777 12092 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:02.108886 12092 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.109061 12092 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:02.109393 11632 ts_manager.cc:194] Registered new tserver with Master: 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:02.109721 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:54987
I20260924 15:28:02.113325 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.98.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.253.keytab.
Entry for principal oryx/127.5.98.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.253.keytab.
I20260924 15:28:02.138998 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:44065
--webserver_interface=127.5.98.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.253.keytab
--principal=oryx/127.5.98.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41153,127.5.98.253:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:02.220412 12097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:02.220639 12097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:02.220676 12097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:02.221802 12097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:02.221863 12097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:02.221895 12097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:02.221925 12097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:02.221954 12097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:02.223469 12097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41153,127.5.98.253:44065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.253.keytab
--principal=oryx/127.5.98.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.98.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
W20260924 15:28:02.223603 12097 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.98.254:41153,127.5.98.253:44065'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 15:28:02.223774 12097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:02.224000 12097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.229051 12105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.229210 12103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.229058 12102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.229176 12097 server_base.cc:1034] running on GCE node
I20260924 15:28:02.229643 12097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.229890 12097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.231046 12097 hybrid_clock.cc:648] HybridClock initialized: now 1790263682231034 us; error 30 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.233603 12097 init.cc:377] Logged in from keytab as oryx/127.5.98.253@KRBTEST.COM (short username oryx)
I20260924 15:28:02.234023 12097 webserver.cc:460] Webserver started at http://127.5.98.253:43593/ using document root <none> and password file <none>
I20260924 15:28:02.234268 12097 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.234335 12097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.234473 12097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:02.235529 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/instance:
uuid: "affd0a0787d248e89c75351151cb11ec"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "308ea994a258bc68b67bd2aaf05e8480"
server_key_iv: "04f086e7681e40976fa4a6c294297787"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.235970 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal/instance:
uuid: "affd0a0787d248e89c75351151cb11ec"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "308ea994a258bc68b67bd2aaf05e8480"
server_key_iv: "04f086e7681e40976fa4a6c294297787"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.237311 12097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:02.238034 12112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.238269 12097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:02.238370 12097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "affd0a0787d248e89c75351151cb11ec"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "308ea994a258bc68b67bd2aaf05e8480"
server_key_iv: "04f086e7681e40976fa4a6c294297787"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.238444 12097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.247635 12097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.249855 12097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:02.250002 12097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.254128 12097 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.253:44065
I20260924 15:28:02.254189 12164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.253:44065 every 8 connection(s)
I20260924 15:28:02.254519 12097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
I20260924 15:28:02.255529 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:02.257285 12165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:28:02.263839 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 12097
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.253@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.271533 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.257922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.5.98.254:41153)
0924 15:28:02.258037 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.258039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.258223 (+ 184us) server_negotiation.cc:407] Connection header received
0924 15:28:02.258828 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.258829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.258844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.258932 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.259593 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.260162 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.261254 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.261465 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.261553 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.268339 (+ 6786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.268341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.268341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.268350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.269942 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.270367 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.270369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.270370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.270435 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.270715 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.270716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.270716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.270876 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.270960 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.271234 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.271319 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 15:28:02.271616 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.257894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40040)
0924 15:28:02.258101 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.258138 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.258685 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.259009 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.259013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.259278 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.259479 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.259483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.260275 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.260276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.261125 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.261127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.261565 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.261567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.261703 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.266697 (+ 4994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.266706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.268244 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.270016 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.270023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.270025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.270266 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.270526 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.270526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.270527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.270647 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.270951 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.270958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.271099 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.271199 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.271383 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260924 15:28:02.272267 12165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:28:02.272804 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.264267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.5.98.253:44065)
0924 15:28:02.264432 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.264435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.264451 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:28:02.264472 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.264474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.264489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.264557 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.265164 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.265892 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.266754 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.267056 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.267137 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.269782 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.269789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.269790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.269795 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.271421 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.271853 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.271857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.271857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.271922 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.272147 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.272148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.272148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.272309 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.272461 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.272555 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.272644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.253@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.278941 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.272526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54416)
0924 15:28:02.272665 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.272675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.272713 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.272955 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.272956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.273094 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.273144 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.273146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.273781 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.273782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.274261 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.274262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.274571 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.274572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.274623 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.274938 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.274942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.276078 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.278079 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.278080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.278080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.278251 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.278429 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.278429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.278430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.278487 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.278689 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.278690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.278724 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.278784 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.278859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260924 15:28:02.279026 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.272558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.5.98.253:44065)
0924 15:28:02.272720 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.272722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.272734 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:02.272834 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.272835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.272860 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.273209 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.273231 (+ 22us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.273719 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.274325 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.274513 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.274560 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.276151 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.276153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.276153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.276159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.277992 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.278321 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.278322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.278322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.278370 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.278542 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.278542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.278543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.278638 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.278685 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.278799 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.278900 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 15:28:02.280323 12165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Bootstrap starting.
I20260924 15:28:02.280992 12165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:02.281270 12165 log.cc:824] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:02.282021 12165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: No bootstrap required, opened a new log
I20260924 15:28:02.283454 12165 raft_consensus.cc:348] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } }
I20260924 15:28:02.283577 12165 raft_consensus.cc:374] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:02.283635 12165 raft_consensus.cc:729] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affd0a0787d248e89c75351151cb11ec, State: Initialized, Role: FOLLOWER
I20260924 15:28:02.283819 12165 consensus_queue.cc:260] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } }
I20260924 15:28:02.284129 12172 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } } }
I20260924 15:28:02.284180 12165 sys_catalog.cc:564] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:02.284235 12172 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:28:02.285879 12183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:28:02.285934 12183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:28:02.287485 12097 master_runner.cc:428] Detected that this master affd0a0787d248e89c75351151cb11ec is joining an existing cluster
I20260924 15:28:02.287523 12097 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.98.253:44065
I20260924 15:28:02.295346 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.288504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.5.98.254:41153)
0924 15:28:02.288547 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.288548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.288672 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:28:02.288779 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.288780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.288794 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.288869 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.289359 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.290078 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.290850 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.291083 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.291142 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.292320 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.292321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.292322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.292330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.293965 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294410 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.294411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.294412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.294470 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294717 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.294718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.294718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.294872 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.294924 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.295167 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.295248 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:28:02.295365 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.288474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40048)
0924 15:28:02.288617 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.288627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.288685 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.288921 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.288923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.289204 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.289277 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.289280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.290152 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.290152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.290769 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.290771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.291207 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.291208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.291271 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.291813 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.291818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.292215 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.294059 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.294061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.294330 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.294545 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.294546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.294619 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.294980 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.294981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.295034 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.295124 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.295239 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":28,"thread_start_us":94,"threads_started":1}
I20260924 15:28:02.295557 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.288548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.5.98.253:44065)
0924 15:28:02.288587 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.288589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.288792 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:28:02.288856 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.288857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.288865 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.288916 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.289281 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.290015 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.291081 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.291366 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.291430 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.292610 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.292611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.292611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.292617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.294192 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294594 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.294595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.294595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.294642 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294838 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.294839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.294840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.294945 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.295013 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.295182 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.295289 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":21}
I20260924 15:28:02.295951 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.288623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54432)
0924 15:28:02.288744 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.288753 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.288802 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.288921 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.288922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.289141 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.289200 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.289202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.290100 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.290101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.290997 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.291000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.291714 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.291715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.291806 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.292178 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.292183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.292535 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.294274 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.294275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.294507 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.294708 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.294709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.294710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.294774 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.295006 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.295006 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.295046 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.295207 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.295430 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.254@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.305078 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.297832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.5.98.253:44065)
0924 15:28:02.297887 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.297889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.297978 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:28:02.298078 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.298079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.298092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.298136 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.298644 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.299048 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.300175 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.300338 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.300388 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.302379 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.302380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.302381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.302387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.303900 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.304325 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.304326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.304326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.304378 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.304575 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.304576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.304576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.304681 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.304745 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.304923 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.304995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:28:02.305104 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.297727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54440)
0924 15:28:02.297873 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.297920 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.297999 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.298158 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.298161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.298500 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.298565 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.298568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.299128 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.299129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.300093 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.300097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.300410 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.300411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.300500 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.300905 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.300910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.302279 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.303985 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.303992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.303994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.304266 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.304449 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.304450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.304450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.304509 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.304751 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.304759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.304806 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.304897 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.305006 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260924 15:28:02.305649 11632 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.98.253:44065: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "affd0a0787d248e89c75351151cb11ec"
member_type: NON_VOTER
last_known_addr {
host: "127.5.98.253"
port: 44065
}
attrs {
promote: true
}
}
dest_uuid: "51cd0b5a18634343ab87689c36f3b53b"
cas_config_opid_index: -1
I20260924 15:28:02.305945 11632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } }
I20260924 15:28:02.306402 12193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER affd0a0787d248e89c75351151cb11ec (127.5.98.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } }
I20260924 15:28:02.306689 12139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:02.307307 12193 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.98.253:44065
I20260924 15:28:02.307518 12097 master.cc:537] Master@127.5.98.253:44065 shutting down...
I20260924 15:28:02.307549 12192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } }
I20260924 15:28:02.307626 12192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:02.307639 12139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 15:28:02.308034 12192 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:28:02.308784 11619 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 15:28:02.310304 12097 raft_consensus.cc:2229] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:02.310393 12097 raft_consensus.cc:2258] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:02.310446 12097 tablet_replica.cc:330] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: stopping tablet replica
I20260924 15:28:02.316015 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.315905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54452)
0924 15:28:02.315950 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.315984 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:28:02.316134 11619 consensus_peers.cc:487] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b -> Peer affd0a0787d248e89c75351151cb11ec (127.5.98.253:44065): Couldn't send request to peer affd0a0787d248e89c75351151cb11ec. Status: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44065: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:28:02.322913 12097 master.cc:552] Master@127.5.98.253:44065 shutdown complete.
I20260924 15:28:02.322971 12097 master_runner.cc:305] Clearing existing system tablet
I20260924 15:28:02.323100 12097 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.323155 12097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.323859 12097 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:02.324393 12197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.324563 12097 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.324620 12097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.324646 12097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "affd0a0787d248e89c75351151cb11ec"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "308ea994a258bc68b67bd2aaf05e8480"
server_key_iv: "04f086e7681e40976fa4a6c294297787"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.325219 12097 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:02.326601 12097 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:28:02.326689 12097 log.cc:1196] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 15:28:02.326881 12097 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Deleting consensus metadata
I20260924 15:28:02.327011 12097 master_runner.cc:315] Copying system tablet from 127.5.98.254:41153
I20260924 15:28:02.327577 12097 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: tablet copy: Beginning tablet copy session from remote peer at address 127.5.98.254:41153
I20260924 15:28:02.334048 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.327958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.5.98.254:41153)
0924 15:28:02.328000 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.328002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.328059 (+ 57us) server_negotiation.cc:407] Connection header received
0924 15:28:02.328139 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.328140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.328149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.328196 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.328896 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.329369 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.330401 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.330575 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.330629 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.331639 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.331640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.331641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.331647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.333005 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.333392 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.333393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.333393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.333441 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.333625 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.333626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.333626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.333720 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.333769 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.333935 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.333991 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:28:02.334132 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.327870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40056)
0924 15:28:02.327991 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.328002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.328071 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.328204 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.328205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.328440 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.328802 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.328805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.329444 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.329445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.330318 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.330321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.330651 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.330652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.330724 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.331152 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.331157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.331556 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.333083 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.333084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.333084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.333315 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.333496 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.333496 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.333497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.333563 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.333782 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.333783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.333830 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.333918 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.334018 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260924 15:28:02.334411 11652 tablet_copy_service.cc:140] P 51cd0b5a18634343ab87689c36f3b53b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer affd0a0787d248e89c75351151cb11ec ({username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:40056)
I20260924 15:28:02.334503 11652 tablet_copy_service.cc:161] P 51cd0b5a18634343ab87689c36f3b53b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer affd0a0787d248e89c75351151cb11ec at {username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:40056: session id = affd0a0787d248e89c75351151cb11ec-00000000000000000000000000000000
I20260924 15:28:02.335083 11652 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:28:02.335705 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:28:02.336568 12097 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: tablet copy: Starting download of 0 data blocks...
I20260924 15:28:02.336709 12097 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: tablet copy: Starting download of 1 WAL segments...
I20260924 15:28:02.337487 12097 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:28:02.338227 11652 tablet_copy_service.cc:342] P 51cd0b5a18634343ab87689c36f3b53b: Request end of tablet copy session affd0a0787d248e89c75351151cb11ec-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:40056
I20260924 15:28:02.338282 11652 tablet_copy_service.cc:433] P 51cd0b5a18634343ab87689c36f3b53b: ending tablet copy session affd0a0787d248e89c75351151cb11ec-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer affd0a0787d248e89c75351151cb11ec
I20260924 15:28:02.340853 12097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.342031 12205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.342202 12204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.342197 12207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.342353 12097 server_base.cc:1034] running on GCE node
I20260924 15:28:02.342458 12097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.342609 12097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.343716 12097 hybrid_clock.cc:648] HybridClock initialized: now 1790263682343712 us; error 20 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.345080 12097 init.cc:377] Logged in from keytab as oryx/127.5.98.253@KRBTEST.COM (short username oryx)
I20260924 15:28:02.345333 12097 webserver.cc:460] Webserver started at http://127.5.98.253:36589/ using document root <none> and password file <none>
I20260924 15:28:02.345453 12097 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.345495 12097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.346210 12097 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:02.346756 12214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.346920 12097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:02.346976 12097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "affd0a0787d248e89c75351151cb11ec"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "308ea994a258bc68b67bd2aaf05e8480"
server_key_iv: "04f086e7681e40976fa4a6c294297787"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.347043 12097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.358000 12097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.359468 12097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.362772 12097 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.253:44065
I20260924 15:28:02.362975 12266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.253:44065 every 8 connection(s)
I20260924 15:28:02.363142 12097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
I20260924 15:28:02.363914 12267 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:28:02.364279 12267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Bootstrap starting.
I20260924 15:28:02.366484 12267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:28:02.366707 12267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Bootstrap complete.
I20260924 15:28:02.366883 12267 raft_consensus.cc:348] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } }
I20260924 15:28:02.366964 12267 raft_consensus.cc:729] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: affd0a0787d248e89c75351151cb11ec, State: Initialized, Role: LEARNER
I20260924 15:28:02.367014 12267 consensus_queue.cc:260] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } }
I20260924 15:28:02.367187 12267 sys_catalog.cc:564] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:02.367170 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } }
I20260924 15:28:02.367254 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: LEARNER
I20260924 15:28:02.368554 12281 catalog_manager.cc:1259] Loaded cluster ID: 661240a6ed4a431e8088fd83cee4e97a
I20260924 15:28:02.368611 12281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: loading cluster ID for follower catalog manager: success
I20260924 15:28:02.369522 12281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: acquiring CA information for follower catalog manager: success
I20260924 15:28:02.369789 12281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:28:02.380771 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.374514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.5.98.253:44065)
0924 15:28:02.374650 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.374653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.374660 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:02.374684 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.374685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.374695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.374779 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.375311 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.376020 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.376725 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.376927 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.376997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.378189 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.378190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.378191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.378198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.379624 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.380000 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.380002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.380002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.380060 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.380256 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.380257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.380257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.380366 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.380466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.380571 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.380655 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.5.98.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.252.keytab.
Entry for principal oryx/127.5.98.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.252.keytab.
I20260924 15:28:02.406674 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:34435
--webserver_interface=127.5.98.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.252.keytab
--principal=oryx/127.5.98.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:02.491292 12285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:02.491492 12285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:02.491514 12285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:02.492686 12285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:02.492724 12285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:02.492738 12285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:02.492779 12285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:02.492801 12285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:02.494234 12285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.252.keytab
--principal=oryx/127.5.98.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.98.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:02.494482 12285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:02.494705 12285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.499563 12293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.499732 12285 server_base.cc:1034] running on GCE node
W20260924 15:28:02.499564 12291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.499800 12290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.500041 12285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.500330 12285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.501468 12285 hybrid_clock.cc:648] HybridClock initialized: now 1790263682501461 us; error 32 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.504066 12285 init.cc:377] Logged in from keytab as oryx/127.5.98.252@KRBTEST.COM (short username oryx)
I20260924 15:28:02.504501 12285 webserver.cc:460] Webserver started at http://127.5.98.252:39493/ using document root <none> and password file <none>
I20260924 15:28:02.504716 12285 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.504760 12285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.504902 12285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:02.505925 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/instance:
uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "51365a1ad007f5689a669fd4599bd8b8"
server_key_iv: "17c3628cf7cd0d2753e1174e2a50612f"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.506285 12285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal/instance:
uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "51365a1ad007f5689a669fd4599bd8b8"
server_key_iv: "17c3628cf7cd0d2753e1174e2a50612f"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.507562 12285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:28:02.508301 12300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.508493 12285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:02.508584 12285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "51365a1ad007f5689a669fd4599bd8b8"
server_key_iv: "17c3628cf7cd0d2753e1174e2a50612f"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.508653 12285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.539695 12285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.542083 12285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:02.542248 12285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.547075 12285 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.252:34435
I20260924 15:28:02.547147 12352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.252:34435 every 8 connection(s)
I20260924 15:28:02.547492 12285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
I20260924 15:28:02.548516 12353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:02.550067 12353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:28:02.551369 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 12285
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.252@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
I20260924 15:28:02.560699 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.550840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.5.98.254:41153)
0924 15:28:02.550906 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.550908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.551012 (+ 104us) server_negotiation.cc:407] Connection header received
0924 15:28:02.551601 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.551602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.551618 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.551730 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.552537 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.553187 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.554568 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.554828 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.554917 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.557416 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.557417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.557418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.557426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.559112 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.559605 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.559606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.559607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.559714 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.559986 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.559987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.559988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.560133 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.560197 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.560530 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.560592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:28:02.560892 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.550694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40070)
0924 15:28:02.550876 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.550923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.551437 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.551819 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.551824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.552197 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.552441 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.552447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.553288 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.553289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.554423 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.554426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.554921 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.554922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.555073 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.555633 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.555646 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.557333 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.559195 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.559200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.559202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.559524 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.559786 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.559787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.559787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.559918 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.560195 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.560200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.560359 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.560475 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.560638 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":177,"thread_start_us":94,"threads_started":1}
I20260924 15:28:02.561327 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.551882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.5.98.252:34435)
0924 15:28:02.552040 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.552044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.552062 (+ 18us) server_negotiation.cc:407] Connection header received
0924 15:28:02.552089 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.552091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.552109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.552179 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.552846 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.553807 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.554720 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.555156 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.555222 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.557995 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.558002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.558003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.558009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.559787 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.560149 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.560151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.560152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.560223 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.560411 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.560412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.560412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.560565 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.560711 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.560901 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.561262 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260924 15:28:02.561437 12353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.252@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.567056 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.561685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54490)
0924 15:28:02.561730 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.561740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.561775 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.561943 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.561943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.562066 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.562097 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.562099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.562690 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.562691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.563167 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.563168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.563476 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.563477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.563517 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.563917 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.563922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.565006 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.566265 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.566266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.566267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.566417 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.566578 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.566578 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.566578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.566627 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.566794 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.566794 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.566827 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.566875 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.566955 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:28:02.567137 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.561767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.5.98.253:44065)
0924 15:28:02.561809 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.561811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.561820 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:02.561848 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.561849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.561861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.561919 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.562158 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.562629 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.563229 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.563415 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.563462 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.565079 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.565080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.565080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.565093 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.566201 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.566476 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.566476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.566477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.566523 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.566665 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.566666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.566666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.566749 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.566792 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.566890 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.567003 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:02.567533 12353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.252@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
I20260924 15:28:02.573498 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.567820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52002)
0924 15:28:02.567868 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.567878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.567915 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.568078 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.568079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.568212 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.568258 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.568260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.568992 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.568993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.569435 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.569436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.569748 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.569749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.569784 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.570074 (+ 290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.570078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.571191 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.572743 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.572744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.572744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.572894 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.573054 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.573054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.573054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.573115 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.573274 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.573275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.573304 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.573350 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.573420 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:28:02.573531 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.567875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.5.98.252:34435)
0924 15:28:02.567908 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.567909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.567918 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:02.567969 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.567970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.567979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.568054 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.568315 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.568913 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.569498 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.569692 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.569739 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.571260 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.571261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.571261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.571267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.572676 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.572954 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.572955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.572955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.573001 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.573152 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.573152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.573153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.573231 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.573273 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.573363 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.573458 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":16}
I20260924 15:28:02.574836 12353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Bootstrap starting.
I20260924 15:28:02.575927 12353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:02.576205 12353 log.cc:824] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:02.576903 12353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: No bootstrap required, opened a new log
I20260924 15:28:02.578274 12353 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } }
I20260924 15:28:02.578415 12353 raft_consensus.cc:374] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:02.578461 12353 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91dafb5c3c34f07b9da5b55b2af386e, State: Initialized, Role: FOLLOWER
I20260924 15:28:02.578622 12353 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } }
I20260924 15:28:02.578919 12353 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:02.578886 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } } }
I20260924 15:28:02.578985 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:28:02.580953 12369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:28:02.581015 12369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:28:02.583184 12285 master_runner.cc:428] Detected that this master a91dafb5c3c34f07b9da5b55b2af386e is joining an existing cluster
I20260924 15:28:02.583237 12285 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.98.252:34435
I20260924 15:28:02.591777 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.585073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52004)
0924 15:28:02.585372 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.585383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.585452 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.585645 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.585645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.585826 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.585875 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.585877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586689 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.586689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.587753 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.587756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588236 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.588309 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.588800 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.588805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.589129 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.590750 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.590752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.590752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.590929 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.591128 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.591129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.591129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.591181 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.591432 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.591432 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.591503 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.591551 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.591620 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":78,"thread_start_us":165,"threads_started":1}
I20260924 15:28:02.591836 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.585008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.5.98.252:34435)
0924 15:28:02.585040 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.585042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.585431 (+ 389us) server_negotiation.cc:407] Connection header received
0924 15:28:02.585555 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.585556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.585565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.585592 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.585946 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586616 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.587847 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588117 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588153 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.589215 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.589217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.589217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.589223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.590675 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.591000 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.591001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.591001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.591061 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.591240 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.591241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.591241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.591367 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.591443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.591601 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.591700 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:28:02.612524 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.584768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.5.98.254:41153)
0924 15:28:02.584815 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.584817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.585016 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:28:02.585141 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.585142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.585155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.585203 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.585791 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586558 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.587851 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588141 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588184 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.608954 (+ 20770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.608956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.608956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.608964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.611101 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611553 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.611555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.611555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.611625 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611870 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.611872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.611872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.612039 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.612102 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.612301 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.612406 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:28:02.612567 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.584765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.5.98.253:44065)
0924 15:28:02.584816 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.584818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.584980 (+ 162us) server_negotiation.cc:407] Connection header received
0924 15:28:02.585083 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.585084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.585095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.585141 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.585782 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586573 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.587849 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588154 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588194 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.608908 (+ 20714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.608910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.608911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.608920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.610920 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611412 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.611414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.611415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.611478 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611957 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.611959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.611959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.612116 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.612181 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.612377 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.612470 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:28:02.612655 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11694
I20260924 15:28:02.612913 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.584741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54506)
0924 15:28:02.584904 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.584915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.585007 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.585236 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.585237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.585597 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.585690 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.585693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586665 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.586665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.587753 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.587756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588261 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.588316 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.608312 (+ 19996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.608317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.608793 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.611018 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.611019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.611337 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.611741 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.611743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.611825 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.612173 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.612174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.612253 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.612693 (+ 440us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.612806 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":75,"thread_start_us":108,"threads_started":1}
I20260924 15:28:02.613499 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.584841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40086)
0924 15:28:02.584955 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.584966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.585044 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.585247 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.585248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.585579 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.585688 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.585690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.586671 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.586672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.587749 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.587752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.588261 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.588262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.588326 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.608257 (+ 19931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.608264 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.608818 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.611200 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.611201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.611478 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.611725 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.611726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.611726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.611796 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.612132 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.612133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.612181 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.613211 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.613341 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":41,"thread_start_us":55,"threads_started":1}
I20260924 15:28:02.620039 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:45047
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:41153
--webserver_port=41761
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263681, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.254@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
I20260924 15:28:02.628528 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.620542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52008)
0924 15:28:02.620589 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.620600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.620639 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.620760 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.620760 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.620907 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.620951 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.620952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.621835 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.621836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.622677 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.622680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.623179 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.623180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.623252 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.623744 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.623750 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.625342 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.627255 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.627257 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.627257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.627488 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.627766 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.627767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.627767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.627835 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.628157 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.628158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.628209 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.628280 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.628391 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:28:02.628861 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.620578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.5.98.252:34435)
0924 15:28:02.620615 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.620617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.620626 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:02.620701 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.620702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.620711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.620758 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.621030 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.621745 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.622767 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.623098 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.623171 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.625442 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.625443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.625444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.625452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.627155 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.627575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.627576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.627577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.627632 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.627908 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.627909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.627909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.628061 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.628126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.628656 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.628737 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:02.629272 11632 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.98.252:34435: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
member_type: NON_VOTER
last_known_addr {
host: "127.5.98.252"
port: 34435
}
attrs {
promote: true
}
}
dest_uuid: "51cd0b5a18634343ab87689c36f3b53b"
cas_config_opid_index: 5
I20260924 15:28:02.629549 11632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } }
I20260924 15:28:02.630081 12192 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.631100 12192 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.98.252:34435
I20260924 15:28:02.631175 12192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.631238 12192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:02.631714 12285 master.cc:537] Master@127.5.98.252:34435 shutting down...
I20260924 15:28:02.631847 12327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:02.632854 12327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:28:02.633407 12193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:28:02.634162 11619 consensus_peers.cc:487] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b -> Peer a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252:34435): Couldn't send request to peer a91dafb5c3c34f07b9da5b55b2af386e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 15:28:02.635039 12285 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:02.635133 12285 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:02.635176 12285 tablet_replica.cc:330] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: stopping tablet replica
I20260924 15:28:02.637775 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.631433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54514)
0924 15:28:02.631605 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.631615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.631651 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.634497 (+ 2846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.634498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.634500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:02.634537 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.634538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.635439 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.635440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.636639 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.636642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.637515 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.637515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.637582 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.637602 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.637667 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.637705 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 15:28:02.637976 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.633034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.5.98.253:44065)
0924 15:28:02.634317 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.634320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.634328 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:02.634353 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.634354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.634377 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.634437 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:02.634616 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.635367 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.636739 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.637351 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.637418 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.637482 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.637866 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.637904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257}
I20260924 15:28:02.638218 12241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 LEARNER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:28:02.638394 12379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 15:28:02.638895 12241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.640300 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cd0b5a18634343ab87689c36f3b53b. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.640389 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: LEARNER
I20260924 15:28:02.640424 12270 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: NON_VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: true } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.640468 12270 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: LEARNER
I20260924 15:28:02.640695 12193 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: attempting to promote NON_VOTER affd0a0787d248e89c75351151cb11ec to VOTER
I20260924 15:28:02.640882 12193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } }
I20260924 15:28:02.641396 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.641304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52012)
0924 15:28:02.641335 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.641370 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 15:28:02.641501 12241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 LEARNER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 15:28:02.641783 12193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 15:28:02.641839 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.641701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52016)
0924 15:28:02.641768 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.641788 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 15:28:02.641966 11619 consensus_peers.cc:487] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b -> Peer a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252:34435): Couldn't send request to peer a91dafb5c3c34f07b9da5b55b2af386e. Status: Network error: Client connection negotiation failed: client connection to 127.5.98.252:34435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:28:02.642472 12379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, affd0a0787d248e89c75351151cb11ec (127.5.98.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.642637 12241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, affd0a0787d248e89c75351151cb11ec (127.5.98.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.643314 12379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.643378 12379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:02.643422 12379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.643460 12379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:02.643611 12381 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.643728 12381 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:28:02.648468 12285 master.cc:552] Master@127.5.98.252:34435 shutdown complete.
I20260924 15:28:02.648514 12285 master_runner.cc:305] Clearing existing system tablet
I20260924 15:28:02.648648 12285 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.648695 12285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.649397 12285 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:02.649922 12383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.650036 12285 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.650081 12285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.650102 12285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "51365a1ad007f5689a669fd4599bd8b8"
server_key_iv: "17c3628cf7cd0d2753e1174e2a50612f"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.650789 12285 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:02.652217 12285 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:28:02.652285 12285 log.cc:1196] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 15:28:02.652467 12285 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Deleting consensus metadata
I20260924 15:28:02.652984 12285 master_runner.cc:315] Copying system tablet from 127.5.98.254:41153
I20260924 15:28:02.653563 12285 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.98.254:41153
I20260924 15:28:02.660596 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.653859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40100)
0924 15:28:02.653988 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.653997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.654053 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.654201 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.654203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.654388 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.654432 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.654434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.655335 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.655336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.656199 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.656202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.656681 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.656682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.656746 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.657274 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.657279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.657661 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.659499 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.659501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.659501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.659751 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.659955 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.659956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.659956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.660020 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.660285 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.660286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.660332 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.660401 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.660500 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260924 15:28:02.660619 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.653917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.5.98.254:41153)
0924 15:28:02.653956 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.653958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.654050 (+ 92us) server_negotiation.cc:407] Connection header received
0924 15:28:02.654105 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.654105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.654116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.654188 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.654509 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.655247 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.656296 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.656595 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.656692 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.657749 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.657750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.657751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.657757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.659410 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.659816 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.659817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.659817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.659877 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.660085 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.660086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.660086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.660217 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.660287 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.660444 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.660521 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:02.660988 11652 tablet_copy_service.cc:140] P 51cd0b5a18634343ab87689c36f3b53b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a91dafb5c3c34f07b9da5b55b2af386e ({username='oryx', principal='oryx/127.5.98.252@KRBTEST.COM'} at 127.0.0.1:40100)
I20260924 15:28:02.661034 11652 tablet_copy_service.cc:161] P 51cd0b5a18634343ab87689c36f3b53b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a91dafb5c3c34f07b9da5b55b2af386e at {username='oryx', principal='oryx/127.5.98.252@KRBTEST.COM'} at 127.0.0.1:40100: session id = a91dafb5c3c34f07b9da5b55b2af386e-00000000000000000000000000000000
I20260924 15:28:02.661597 11652 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:28:02.662045 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:28:02.663249 12285 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: tablet copy: Starting download of 0 data blocks...
I20260924 15:28:02.663388 12285 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: tablet copy: Starting download of 1 WAL segments...
I20260924 15:28:02.664095 12285 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:28:02.664908 11652 tablet_copy_service.cc:342] P 51cd0b5a18634343ab87689c36f3b53b: Request end of tablet copy session a91dafb5c3c34f07b9da5b55b2af386e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.98.252@KRBTEST.COM'} at 127.0.0.1:40100
I20260924 15:28:02.664964 11652 tablet_copy_service.cc:433] P 51cd0b5a18634343ab87689c36f3b53b: ending tablet copy session a91dafb5c3c34f07b9da5b55b2af386e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a91dafb5c3c34f07b9da5b55b2af386e
I20260924 15:28:02.667490 12285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.668877 12390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.672430 12391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.672524 12393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.673012 12285 server_base.cc:1034] running on GCE node
I20260924 15:28:02.673200 12285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.673331 12285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.674446 12285 hybrid_clock.cc:648] HybridClock initialized: now 1790263682674438 us; error 27 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.676061 12285 init.cc:377] Logged in from keytab as oryx/127.5.98.252@KRBTEST.COM (short username oryx)
I20260924 15:28:02.676308 12285 webserver.cc:460] Webserver started at http://127.5.98.252:45199/ using document root <none> and password file <none>
I20260924 15:28:02.676414 12285 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.676440 12285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.677208 12285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:02.677831 12400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.678005 12285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.678052 12285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "51365a1ad007f5689a669fd4599bd8b8"
server_key_iv: "17c3628cf7cd0d2753e1174e2a50612f"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.678103 12285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.720122 12285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.722081 12285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.725142 12285 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.252:34435
I20260924 15:28:02.725548 12285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
I20260924 15:28:02.725551 12452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.252:34435 every 8 connection(s)
I20260924 15:28:02.726466 12453 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:28:02.726964 12453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Bootstrap starting.
I20260924 15:28:02.729993 12453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:28:02.730213 12453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: Bootstrap complete.
I20260924 15:28:02.730400 12453 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } }
I20260924 15:28:02.730484 12453 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a91dafb5c3c34f07b9da5b55b2af386e, State: Initialized, Role: LEARNER
I20260924 15:28:02.730517 12453 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } }
I20260924 15:28:02.730690 12453 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:02.730676 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:02.730792 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: LEARNER
W20260924 15:28:02.730646 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:02.730881 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:02.730928 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 15:28:02.732196 12470 catalog_manager.cc:1259] Loaded cluster ID: 661240a6ed4a431e8088fd83cee4e97a
I20260924 15:28:02.732246 12470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: loading cluster ID for follower catalog manager: success
W20260924 15:28:02.733001 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:02.733043 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:02.733071 12378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:02.733269 12470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: acquiring CA information for follower catalog manager: success
I20260924 15:28:02.733541 12470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:28:02.734709 12378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:45047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=41761
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:02.734952 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:02.735198 12378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.740260 12472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.740243 12475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.740235 12473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.740520 12378 server_base.cc:1034] running on GCE node
I20260924 15:28:02.740737 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.740993 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.742136 12378 hybrid_clock.cc:648] HybridClock initialized: now 1790263682742126 us; error 27 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.744616 12378 init.cc:377] Logged in from keytab as oryx/127.5.98.193@KRBTEST.COM (short username oryx)
I20260924 15:28:02.745039 12378 webserver.cc:460] Webserver started at http://127.5.98.193:41761/ using document root <none> and password file <none>
I20260924 15:28:02.745229 12378 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.745278 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.746531 12378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:02.747044 12482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.747213 12378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:02.747298 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "f52f867273704793b3a3b4aca920afcf"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "1deb24b15b26fe2d6c20fdcf3e375d50"
server_key_iv: "cc302ac49f53b600deb4eeffe514cfc2"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.747637 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.759516 12378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.761385 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:02.761528 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.761726 12378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:02.762009 12378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:02.762041 12378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.762061 12378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:02.762110 12378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.768383 12378 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:45047
I20260924 15:28:02.768447 12595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:45047 every 8 connection(s)
I20260924 15:28:02.768836 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:02.775314 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 12378
I20260924 15:28:02.775484 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11828
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.780633 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.771001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:56149 (local address 127.5.98.253:44065)
0924 15:28:02.771055 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.771057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.771303 (+ 246us) server_negotiation.cc:407] Connection header received
0924 15:28:02.771355 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.771356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.771367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.771432 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.772008 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772750 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.773772 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.773980 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774061 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.777132 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.777134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.777135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.777143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.778949 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.779444 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.779446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.779446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.779508 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.779840 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.779842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.779842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.780004 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.780048 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.780412 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.780499 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:02.781252 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.770913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.5.98.193:56149)
0924 15:28:02.771227 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.771237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.771296 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.771449 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.771449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.771769 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.771900 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.771902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772850 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.772851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.773677 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.773679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.774409 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.774513 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.775238 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.775242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.777037 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.779024 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.779031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.779034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.779365 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.779586 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.779587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.779588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.779757 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.780104 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.780110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.780266 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.780759 (+ 493us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.780953 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":5,"thread_start_us":140,"threads_started":1}
I20260924 15:28:02.781869 12598 heartbeater.cc:347] Connected to a master server at 127.5.98.253:44065
I20260924 15:28:02.782025 12598 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.782251 12598 heartbeater.cc:511] Master 127.5.98.253:44065 requested a full tablet report, sending...
I20260924 15:28:02.782423 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:37111
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:41153
--webserver_port=33047
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:28:02.782771 12231 ts_manager.cc:194] Registered new tserver with Master: f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
I20260924 15:28:02.785204 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.769565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:55697 (local address 127.5.98.254:41153)
0924 15:28:02.769624 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.769626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.769996 (+ 370us) server_negotiation.cc:407] Connection header received
0924 15:28:02.770718 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.770720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.770733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.770791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.771973 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772758 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774097 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.774303 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774376 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.781888 (+ 7512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.781890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.781890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.781900 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.783459 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.783868 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.783870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.783870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.783935 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.784138 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.784139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.784139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.784276 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.784327 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.784818 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.785115 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:28:02.785465 12600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.769565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:55697)
0924 15:28:02.769832 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.769874 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.770612 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.770861 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.770865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.771630 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.771882 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.771887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772849 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.772850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.773986 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.773988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.774374 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.774529 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.780352 (+ 5823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.780356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.781673 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.783535 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.783536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.783536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.783764 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.784010 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.784011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.784011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.784069 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.784361 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.784362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.784673 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.785299 (+ 626us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.785396 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260924 15:28:02.785869 12596 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:02.785923 12596 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.785996 12596 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:02.786244 11632 ts_manager.cc:194] Re-registered known tserver with Master: f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
I20260924 15:28:02.786675 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:55697
I20260924 15:28:02.787506 12602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.769850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.5.98.193:33047)
0924 15:28:02.770063 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.770073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.770620 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.771084 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.771085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.771631 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.771887 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.771890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772849 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.772850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.773946 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.773964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.774365 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.774513 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.775154 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.775162 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.784084 (+ 8922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.786118 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.786119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.786120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.786343 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.786575 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.786575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.786576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.786639 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.786916 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.786917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.787267 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.787342 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.787422 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":48,"thread_start_us":151,"threads_started":1}
I20260924 15:28:02.787628 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.769611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:33047 (local address 127.5.98.252:34435)
0924 15:28:02.770850 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.770852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.770861 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:02.770906 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.770908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.770920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.771006 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.772000 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.772651 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774051 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.774279 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.774356 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.784186 (+ 9830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.784187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.784187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.784195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.786010 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.786432 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.786433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.786434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.786500 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.786707 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.786708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.786708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.786854 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.786926 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.787395 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.787495 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":70,"threads_started":1}
I20260924 15:28:02.788050 12597 heartbeater.cc:347] Connected to a master server at 127.5.98.252:34435
I20260924 15:28:02.788117 12597 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.788194 12597 heartbeater.cc:511] Master 127.5.98.252:34435 requested a full tablet report, sending...
I20260924 15:28:02.788528 12417 ts_manager.cc:194] Registered new tserver with Master: f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
W20260924 15:28:02.866690 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:02.866902 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:02.866938 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:02.868309 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:02.868381 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:02.868425 12604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:02.869967 12604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:37111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=33047
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:02.870232 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:02.870471 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:02.875372 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:02.875536 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.875500 12604 server_base.cc:1034] running on GCE node
W20260924 15:28:02.875734 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:02.875988 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:02.876217 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:02.877380 12604 hybrid_clock.cc:648] HybridClock initialized: now 1790263682877373 us; error 35 us; skew 500 ppm
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:02.879906 12604 init.cc:377] Logged in from keytab as oryx/127.5.98.194@KRBTEST.COM (short username oryx)
I20260924 15:28:02.880302 12604 webserver.cc:460] Webserver started at http://127.5.98.194:33047/ using document root <none> and password file <none>
I20260924 15:28:02.880546 12604 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:02.880622 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:02.881937 12604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:02.882469 12619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.882642 12604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:02.882737 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "783574c66bc04c82a98647ad9e156ba6"
format_stamp: "Formatted at 2026-09-24 15:28:01 on dist-test-slave-4g6f"
server_key: "b5808dfb6b898b192460e280d4757eae"
server_key_iv: "80057fff9542baa7964d99c49d1c0232"
server_key_version: "encryptionkey@0"
I20260924 15:28:02.883061 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:02.891004 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:02.893211 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:02.893361 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:02.893572 12604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:02.893857 12604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:02.893903 12604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.893949 12604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:02.893978 12604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:02.899619 12604 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:37111
I20260924 15:28:02.899739 12732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:37111 every 8 connection(s)
I20260924 15:28:02.900034 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
Sep 24 15:28:02 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263682, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:02.908346 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 12604
I20260924 15:28:02.908500 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11962
I20260924 15:28:02.910702 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.900790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:43815 (local address 127.5.98.254:41153)
0924 15:28:02.900836 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.900838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.901217 (+ 379us) server_negotiation.cc:407] Connection header received
0924 15:28:02.901684 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.901686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.901700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.901751 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.902507 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903053 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904213 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904466 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904523 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.907087 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.907088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.907088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.907094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.909071 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.909571 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.909572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.909573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.909635 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.909915 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.909916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.909916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.910064 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.910129 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.910496 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.910581 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:28:02.910897 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.901081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:41729 (local address 127.5.98.253:44065)
0924 15:28:02.901141 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.901143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.901367 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:28:02.901700 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.901701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.901714 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.901748 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.902545 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903279 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904424 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904627 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904697 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.908078 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.908079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.908080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.908095 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.909748 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.910084 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.910085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.910086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.910140 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.910331 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.910332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.910332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.910465 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.910527 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.910727 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.910812 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:28:02.910931 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.900733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:43815)
0924 15:28:02.901143 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.901152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.901579 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.901820 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.901824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.902215 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.902403 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.902408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903139 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.903140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904117 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.904119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904584 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904726 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.905406 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.905411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.906982 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.909170 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.909177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.909179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.909474 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.909706 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.909706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.909707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.909846 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.910157 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.910164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.910341 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.910463 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.910648 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260924 15:28:02.911361 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.901003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.5.98.194:41729)
0924 15:28:02.901299 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.901319 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.901588 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.901824 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.901824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.902309 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.902404 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.902411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903374 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.903375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904318 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.904321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904708 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904777 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.905356 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.905365 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.907989 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.909829 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.909829 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.909830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.910016 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.910210 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.910211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.910211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.910265 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.910555 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.910556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.910605 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.911163 (+ 558us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.911261 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":26,"thread_start_us":61,"threads_started":1}
I20260924 15:28:02.911795 12733 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:02.911931 12733 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.912016 12735 heartbeater.cc:347] Connected to a master server at 127.5.98.253:44065
I20260924 15:28:02.912077 12735 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.912173 12733 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:02.912439 12735 heartbeater.cc:511] Master 127.5.98.253:44065 requested a full tablet report, sending...
I20260924 15:28:02.912853 11632 ts_manager.cc:194] Re-registered known tserver with Master: 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111)
I20260924 15:28:02.912887 12231 ts_manager.cc:194] Registered new tserver with Master: 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111)
I20260924 15:28:02.913381 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:43815
I20260924 15:28:02.914196 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.900714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.5.98.194:33261)
0924 15:28:02.900881 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:02.900921 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:02.901575 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:02.901824 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:02.901824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:02.902214 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:02.902404 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.902411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903446 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.903446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904318 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:02.904320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904784 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:02.904844 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.905496 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:02.905500 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:02.907474 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:02.913063 (+ 5589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.913064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.913064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.913269 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.913494 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:02.913495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:02.913496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:02.913554 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:02.913858 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:02.913859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:02.913914 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 15:28:02.913987 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:28:02.914080 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":24,"thread_start_us":95,"threads_started":1}
I20260924 15:28:02.914279 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:02.900737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:33261 (local address 127.5.98.252:34435)
0924 15:28:02.900793 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:02.900796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:02.901016 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:28:02.901701 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:02.901702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:02.901717 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:02.901748 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:02.902516 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.903305 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904403 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:02.904711 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:02.904784 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:02.907575 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:02.907576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:02.907576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:02.907584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:02.912984 (+ 5400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.913342 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.913344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.913344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.913401 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:02.913636 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:02.913636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:02.913637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:02.913764 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:02.913824 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:02.914036 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 15:28:02.914140 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:28:02.915580 12734 heartbeater.cc:347] Connected to a master server at 127.5.98.252:34435
I20260924 15:28:02.915635 12734 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:02.915742 12734 heartbeater.cc:511] Master 127.5.98.252:34435 requested a full tablet report, sending...
I20260924 15:28:02.915896 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:39463
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:41153
--webserver_port=35435
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:28:02.917409 12417 ts_manager.cc:194] Registered new tserver with Master: 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111)
W20260924 15:28:02.999866 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:03.000062 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:03.000085 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:03.001245 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:03.001286 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:03.001336 12740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:03.002847 12740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:39463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=35435
--tserver_master_addrs=127.5.98.254:41153,127.5.98.253:44065,127.5.98.252:34435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:03.003087 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:03.003319 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:03.008797 12746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:03.008922 12745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:03.009002 12748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:03.009054 12740 server_base.cc:1034] running on GCE node
I20260924 15:28:03.009254 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:03.009532 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:03.010699 12740 hybrid_clock.cc:648] HybridClock initialized: now 1790263683010676 us; error 39 us; skew 500 ppm
Sep 24 15:28:03 dist-test-slave-4g6f krb5kdc[11587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:03.013255 12740 init.cc:377] Logged in from keytab as oryx/127.5.98.195@KRBTEST.COM (short username oryx)
I20260924 15:28:03.013721 12740 webserver.cc:460] Webserver started at http://127.5.98.195:35435/ using document root <none> and password file <none>
I20260924 15:28:03.013950 12740 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:03.014029 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:03.015363 12740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:03.015981 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:03.016178 12740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:03.016268 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "9269a6f59fe94cfc99757bc2276b1a2d"
format_stamp: "Formatted at 2026-09-24 15:28:02 on dist-test-slave-4g6f"
server_key: "ae3b4aa3c1d74d6e2aa90e8d7efd499e"
server_key_iv: "71e241384e2a181cb3bfd8d2c86b11c4"
server_key_version: "encryptionkey@0"
I20260924 15:28:03.016623 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:03.024245 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:03.026449 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:03.026558 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:03.026744 12740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:03.026993 12740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:03.027024 12740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:03.027045 12740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:03.027057 12740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:03.033257 12740 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:39463
I20260924 15:28:03.033313 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:39463 every 8 connection(s)
I20260924 15:28:03.033643 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:28:03 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
Sep 24 15:28:03 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.252@KRBTEST.COM
I20260924 15:28:03.040545 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 12740
Sep 24 15:28:03 dist-test-slave-4g6f krb5kdc[11587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263683, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.253@KRBTEST.COM
I20260924 15:28:03.044993 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:56507 (local address 127.5.98.254:41153)
0924 15:28:03.034496 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.034498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.034701 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:28:03.035300 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.035301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.035315 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.035347 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.036107 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036608 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.037584 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.037801 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.037874 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.040545 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.040547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.040547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.040555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.042548 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.043499 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.043500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.043501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.043561 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.043889 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.043890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.043891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.044086 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.044135 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.044790 (+ 655us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.044868 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 15:28:03.046085 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:45469 (local address 127.5.98.252:34435)
0924 15:28:03.034454 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.034456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.034644 (+ 188us) server_negotiation.cc:407] Connection header received
0924 15:28:03.035254 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.035255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.035269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.035301 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.036121 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036886 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038042 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.038272 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038303 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.041654 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.041655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.041656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.041663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.043793 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.045329 (+ 1536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.045330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.045330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.045384 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.045565 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.045566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.045566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.045705 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.045742 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.045915 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.045997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 15:28:03.046558 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.5.98.195:45469)
0924 15:28:03.034515 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.034556 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.035155 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.035391 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.035395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.035883 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:03.036013 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.036020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036976 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.036977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.037911 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.037928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.038364 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.038408 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.038732 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:03.038735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:03.041553 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:03.044237 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.044237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.044238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.045255 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.045458 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.045458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.045459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.045508 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.045783 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:03.045783 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:03.045821 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.046160 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.046328 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":30,"thread_start_us":88,"threads_started":1}
I20260924 15:28:03.047004 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:46533 (local address 127.5.98.253:44065)
0924 15:28:03.034806 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.034808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.034928 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:28:03.035245 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.035247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.035259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.035318 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.036132 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036876 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038091 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.038895 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038969 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.042765 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.042766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.042767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.042774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.045085 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.045982 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.045983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.045984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.046047 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.046242 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.046243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.046243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.046364 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.046435 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.046799 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.046909 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:28:03.047351 12870 heartbeater.cc:347] Connected to a master server at 127.5.98.252:34435
I20260924 15:28:03.047451 12870 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:03.047647 12870 heartbeater.cc:511] Master 127.5.98.252:34435 requested a full tablet report, sending...
I20260924 15:28:03.047818 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.5.98.195:46533)
0924 15:28:03.034859 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.034868 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.035155 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.035391 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.035395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.035850 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:03.036013 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.036020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036976 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.036977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.038012 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.038014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.038983 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.038984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.039137 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.039551 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:03.039555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:03.042695 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:03.045699 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.045700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.045701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.045905 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.046122 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.046123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.046123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.046177 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.046472 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:03.046472 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:03.046511 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.046571 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.047392 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260924 15:28:03.048434 12417 ts_manager.cc:194] Registered new tserver with Master: 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:03.049343 12871 heartbeater.cc:347] Connected to a master server at 127.5.98.253:44065
I20260924 15:28:03.049397 12871 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:03.049741 12871 heartbeater.cc:511] Master 127.5.98.253:44065 requested a full tablet report, sending...
I20260924 15:28:03.050285 12231 ts_manager.cc:194] Registered new tserver with Master: 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:03.051002 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.034351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:56507)
0924 15:28:03.034545 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.034558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.035143 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.035422 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.035423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.035812 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:03.036011 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.036016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.036711 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.036711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.037494 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.037496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.037881 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.037882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.038033 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.038568 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:03.038577 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:03.040441 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:03.042851 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.042858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.042860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.043403 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.043637 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:03.043637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:03.043638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:03.043792 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:03.044191 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:03.044436 (+ 245us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:03.044604 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.047502 (+ 2898us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.047643 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260924 15:28:03.051316 12869 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41153
I20260924 15:28:03.051373 12869 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:03.051457 12869 heartbeater.cc:511] Master 127.5.98.254:41153 requested a full tablet report, sending...
I20260924 15:28:03.051890 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.042493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.5.98.254:41153)
0924 15:28:03.043793 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.043795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.043802 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:03.043834 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.043835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.043847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.044491 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.044525 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.045263 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.046291 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.046606 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.046950 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.048660 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.048661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.048662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.048669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.050295 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.050716 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.050717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.050717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.050777 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.050998 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.050999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.050999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.051137 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.051637 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.051699 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.051784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":316,"threads_started":1}
I20260924 15:28:03.052028 11632 ts_manager.cc:194] Re-registered known tserver with Master: 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:03.052670 11632 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:56507
I20260924 15:28:03.052897 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.042256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.5.98.253:44065)
0924 15:28:03.042956 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.042958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.042966 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:03.042986 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.042987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.042997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.043704 (+ 707us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.043912 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.044685 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.047487 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.047797 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.047863 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.049650 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.049652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.049652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.049659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.051407 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.051820 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.051821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.051821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.051877 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.052077 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.052078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.052078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.052237 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.052325 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.052703 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.052792 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":99,"threads_started":1}
I20260924 15:28:03.053707 11632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 15:28:03.055565 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.042118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.5.98.252:34435)
0924 15:28:03.042808 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.042811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.042817 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:28:03.042841 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.042841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.042853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.042913 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.048379 (+ 5466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.049137 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.050553 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.050887 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.050956 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.053143 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.053144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.053144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.053151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.054542 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.054901 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.054902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.054902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.054950 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.055149 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.055149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.055150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.055239 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.055327 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.055449 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.055506 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":64,"threads_started":1}
W20260924 15:28:03.055747 11632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:03.062942 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:45047 (local address 127.0.0.1:38690)
0924 15:28:03.059102 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.059113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.059152 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.059876 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.059877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.059878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.059926 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.059928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.060831 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.060832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.061748 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.061750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.062694 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.062762 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.062766 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.062820 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.062859 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 15:28:03.063067 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:39463 (local address 127.0.0.1:33898)
0924 15:28:03.059358 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.059369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.059424 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.059818 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.059819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.059821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.059863 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.059864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.061168 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.061169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.062154 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.062156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.062817 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.062877 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.062881 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.062950 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.062987 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260924 15:28:03.063359 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.5.98.193:45047)
0924 15:28:03.059620 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.059625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.059637 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:03.059728 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.059730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.059747 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.059828 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.059991 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.060744 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.061844 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.062604 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062680 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.062764 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.063231 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.063278 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":117,"threads_started":1}
I20260924 15:28:03.063385 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.5.98.195:39463)
0924 15:28:03.059540 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.059545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.059560 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:28:03.059589 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.059590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.059603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.059758 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.059961 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.061069 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062229 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.062717 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.062871 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.062965 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.063326 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260924 15:28:03.063827 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:37111 (local address 127.0.0.1:38236)
0924 15:28:03.059537 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.059547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.059632 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.060273 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.060274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.060276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.060312 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.060314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.061596 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.061598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.062668 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.062671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.063575 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.063577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.063638 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.063643 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.063721 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.063760 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":325,"thread_start_us":72,"threads_started":1}
I20260924 15:28:03.063846 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.059251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.5.98.194:37111)
0924 15:28:03.060020 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.060025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.060034 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:03.060109 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.060111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.060127 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.060202 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.060383 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.061513 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.062756 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.063474 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.063571 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.063641 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.063735 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.063771 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":79,"threads_started":1}
I20260924 15:28:03.064116 12530 tablet_service.cc:1467] Processing CreateTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 (DEFAULT_TABLE table=test-table [id=bfcbe185b0aa429086d40338dd72153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:03.064129 12803 tablet_service.cc:1467] Processing CreateTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 (DEFAULT_TABLE table=test-table [id=bfcbe185b0aa429086d40338dd72153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:03.064465 12530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c90b6c4910b4d9788cfc81d4aca08c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:03.064465 12803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c90b6c4910b4d9788cfc81d4aca08c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:03.064435 12667 tablet_service.cc:1467] Processing CreateTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 (DEFAULT_TABLE table=test-table [id=bfcbe185b0aa429086d40338dd72153d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:03.064683 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c90b6c4910b4d9788cfc81d4aca08c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:03.067303 12895 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Bootstrap starting.
I20260924 15:28:03.068073 12895 tablet_bootstrap.cc:654] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:03.068470 12895 log.cc:824] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:03.068487 12896 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Bootstrap starting.
I20260924 15:28:03.068836 12897 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Bootstrap starting.
I20260924 15:28:03.069268 12896 tablet_bootstrap.cc:654] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:03.069388 12897 tablet_bootstrap.cc:654] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:03.069459 12895 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: No bootstrap required, opened a new log
I20260924 15:28:03.069533 12895 ts_tablet_manager.cc:1397] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:03.069576 12896 log.cc:824] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:03.070078 12897 log.cc:824] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:03.070714 12896 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: No bootstrap required, opened a new log
I20260924 15:28:03.070741 12897 tablet_bootstrap.cc:492] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: No bootstrap required, opened a new log
I20260924 15:28:03.070785 12896 ts_tablet_manager.cc:1397] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:03.070814 12897 ts_tablet_manager.cc:1397] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:28:03.071524 12895 raft_consensus.cc:348] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.071771 12895 raft_consensus.cc:374] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:03.071820 12895 raft_consensus.cc:729] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783574c66bc04c82a98647ad9e156ba6, State: Initialized, Role: FOLLOWER
I20260924 15:28:03.071973 12895 consensus_queue.cc:260] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.072247 12895 ts_tablet_manager.cc:1428] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:28:03.072360 12733 heartbeater.cc:503] Master 127.5.98.254:41153 was elected leader, sending a full tablet report...
I20260924 15:28:03.073680 12897 raft_consensus.cc:348] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.073866 12897 raft_consensus.cc:374] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:03.073922 12897 raft_consensus.cc:729] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52f867273704793b3a3b4aca920afcf, State: Initialized, Role: FOLLOWER
I20260924 15:28:03.074043 12897 consensus_queue.cc:260] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.074316 12596 heartbeater.cc:503] Master 127.5.98.254:41153 was elected leader, sending a full tablet report...
I20260924 15:28:03.074469 12897 ts_tablet_manager.cc:1428] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 15:28:03.074859 12896 raft_consensus.cc:348] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.075026 12896 raft_consensus.cc:374] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:03.075064 12896 raft_consensus.cc:729] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269a6f59fe94cfc99757bc2276b1a2d, State: Initialized, Role: FOLLOWER
I20260924 15:28:03.075201 12896 consensus_queue.cc:260] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.075443 12896 ts_tablet_manager.cc:1428] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 15:28:03.075502 12869 heartbeater.cc:503] Master 127.5.98.254:41153 was elected leader, sending a full tablet report...
I20260924 15:28:03.086441 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.083954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.5.98.252:34435)
0924 15:28:03.084009 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.084011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.084035 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:28:03.084059 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.084060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.084086 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.084151 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.084260 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.084769 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.085658 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.086120 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.086202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.086275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.086334 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.086372 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:28:03.086441 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.083902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52042)
0924 15:28:03.083951 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.083961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.084003 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.084153 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.084154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.084156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.084202 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.084203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.084849 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.084850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.085554 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.085555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.086203 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.086204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.086264 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.086267 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.086328 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.086372 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 15:28:03.086840 12427 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 15:28:03.087697 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cd0b5a18634343ab87689c36f3b53b. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:03.087800 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: LEARNER
W20260924 15:28:03.151010 12736 tablet.cc:2369] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:03.156059 12901 raft_consensus.cc:482] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:03.156172 12901 raft_consensus.cc:504] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.156563 12901 leader_election.cc:288] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047), 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:03.159426 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.156798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:48269 (local address 127.5.98.193:45047)
0924 15:28:03.156870 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.156872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.156884 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:03.156953 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.156954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.156968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.157026 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.157221 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.157713 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.158720 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.159068 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.159148 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.159157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.159312 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.159352 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:28:03.159440 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.156799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:57447 (local address 127.5.98.195:39463)
0924 15:28:03.156862 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.156864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.156976 (+ 112us) server_negotiation.cc:407] Connection header received
0924 15:28:03.157041 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.157042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.157055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.157109 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:03.157273 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.157738 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.158722 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.159068 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.159152 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.159160 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.159337 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.159377 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:28:03.159440 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.156728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:45047 (local address 127.5.98.194:48269)
0924 15:28:03.156793 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.156802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.156859 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.157038 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.157039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.157041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.157148 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.157150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.157788 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.157789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.158633 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.158635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.159144 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.159145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.159244 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.159247 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.159289 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.159346 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":30}
I20260924 15:28:03.159463 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.156728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:39463 (local address 127.5.98.194:57447)
0924 15:28:03.156904 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:03.156916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:03.156977 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:03.157128 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:03.157129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:03.157130 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:03.157199 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.157201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.157836 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.157837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.158637 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:03.158638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.159152 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.159153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:03.159250 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.159253 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:03.159324 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:28:03.159358 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":167}
I20260924 15:28:03.159824 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c90b6c4910b4d9788cfc81d4aca08c2" candidate_uuid: "783574c66bc04c82a98647ad9e156ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52f867273704793b3a3b4aca920afcf" is_pre_election: true
I20260924 15:28:03.159988 12550 raft_consensus.cc:2454] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783574c66bc04c82a98647ad9e156ba6 in term 0.
I20260924 15:28:03.159958 12823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c90b6c4910b4d9788cfc81d4aca08c2" candidate_uuid: "783574c66bc04c82a98647ad9e156ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" is_pre_election: true
I20260924 15:28:03.160079 12823 raft_consensus.cc:2454] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783574c66bc04c82a98647ad9e156ba6 in term 0.
I20260924 15:28:03.160238 12623 leader_election.cc:302] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783574c66bc04c82a98647ad9e156ba6, f52f867273704793b3a3b4aca920afcf; no voters:
I20260924 15:28:03.160362 12901 raft_consensus.cc:2781] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:03.160414 12901 raft_consensus.cc:482] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:03.160463 12901 raft_consensus.cc:3037] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:03.161159 12901 raft_consensus.cc:504] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.161294 12901 leader_election.cc:288] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [CANDIDATE]: Term 1 election: Requested vote from peers f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047), 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463)
I20260924 15:28:03.161450 12823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c90b6c4910b4d9788cfc81d4aca08c2" candidate_uuid: "783574c66bc04c82a98647ad9e156ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9269a6f59fe94cfc99757bc2276b1a2d"
I20260924 15:28:03.161450 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c90b6c4910b4d9788cfc81d4aca08c2" candidate_uuid: "783574c66bc04c82a98647ad9e156ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52f867273704793b3a3b4aca920afcf"
I20260924 15:28:03.161527 12550 raft_consensus.cc:3037] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:03.161527 12823 raft_consensus.cc:3037] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:03.162395 12550 raft_consensus.cc:2454] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783574c66bc04c82a98647ad9e156ba6 in term 1.
I20260924 15:28:03.162387 12823 raft_consensus.cc:2454] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783574c66bc04c82a98647ad9e156ba6 in term 1.
I20260924 15:28:03.162569 12621 leader_election.cc:302] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783574c66bc04c82a98647ad9e156ba6, 9269a6f59fe94cfc99757bc2276b1a2d; no voters:
I20260924 15:28:03.162652 12901 raft_consensus.cc:2781] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:03.162799 12901 raft_consensus.cc:686] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 783574c66bc04c82a98647ad9e156ba6, State: Running, Role: LEADER
I20260924 15:28:03.162894 12901 consensus_queue.cc:237] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:03.163687 11632 catalog_manager.cc:5495] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194). New cstate: current_term: 1 leader_uuid: "783574c66bc04c82a98647ad9e156ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } health_report { overall_health: UNKNOWN } } }
I20260924 15:28:03.164244 12379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: NON_VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: true } } }
I20260924 15:28:03.164337 12379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:03.164927 12192 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: attempting to promote NON_VOTER a91dafb5c3c34f07b9da5b55b2af386e to VOTER
I20260924 15:28:03.165128 12192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } }
I20260924 15:28:03.165722 12427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 LEARNER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 15:28:03.165746 12241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Refusing update from remote peer 51cd0b5a18634343ab87689c36f3b53b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 15:28:03.165933 12379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 15:28:03.166028 12911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [LEADER]: Connected to new peer: Peer: permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 15:28:03.166658 12379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.166891 12427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.166965 12241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.167560 12379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.167600 12193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.167639 12379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:03.167708 12193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:03.167752 12193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.167810 12193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:03.167960 12912 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.168056 12912 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:28:03.168061 12459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51cd0b5a18634343ab87689c36f3b53b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:03.168145 12459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:28:03.181041 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.178519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.5.98.194:37111)
0924 15:28:03.178578 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.178580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.178602 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:28:03.178701 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.178702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.178722 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.178805 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:03.178976 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.179455 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.180262 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.180469 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.180550 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.180722 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:03.180830 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:03.180920 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.180939 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.180960 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:28:03.182756 12823 raft_consensus.cc:1261] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Refusing update from remote peer 783574c66bc04c82a98647ad9e156ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:03.182796 12550 raft_consensus.cc:1261] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Refusing update from remote peer 783574c66bc04c82a98647ad9e156ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:03.183111 12901 consensus_queue.cc:1035] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:03.183248 12908 consensus_queue.cc:1035] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:03.186323 12915 mvcc.cc:205] Tried to move back new op lower bound from 7332920046314037248 to 7332920046235799552. Current Snapshot: MvccSnapshot[applied={T|T < 7332920046314037248}]
I20260924 15:28:03.188630 12916 mvcc.cc:205] Tried to move back new op lower bound from 7332920046314037248 to 7332920046235799552. Current Snapshot: MvccSnapshot[applied={T|T < 7332920046314037248}]
I20260924 15:28:03.189980 11632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:03.190069 11632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:03.191254 11632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51cd0b5a18634343ab87689c36f3b53b: Sending DeleteTablet for 3 replicas of tablet 2c90b6c4910b4d9788cfc81d4aca08c2
I20260924 15:28:03.191574 12530 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:03 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:38690
I20260924 15:28:03.191605 12803 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:03 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:33898
I20260924 15:28:03.192229 12925 tablet_replica.cc:330] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: stopping tablet replica
I20260924 15:28:03.192225 12667 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c90b6c4910b4d9788cfc81d4aca08c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:03 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:38236
I20260924 15:28:03.192340 12925 raft_consensus.cc:2229] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:03.192451 12925 raft_consensus.cc:2258] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:03.192466 12924 tablet_replica.cc:330] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: stopping tablet replica
I20260924 15:28:03.192683 12926 tablet_replica.cc:330] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: stopping tablet replica
I20260924 15:28:03.192767 12926 raft_consensus.cc:2229] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:03.192854 12926 raft_consensus.cc:2258] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:03.192915 12925 ts_tablet_manager.cc:1905] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:03.192991 12924 raft_consensus.cc:2229] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:03.193171 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 11600
I20260924 15:28:03.195292 12925 ts_tablet_manager.cc:1918] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:03.195361 12925 log.cc:1196] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/2c90b6c4910b4d9788cfc81d4aca08c2
I20260924 15:28:03.195647 12925 ts_tablet_manager.cc:1939] T 2c90b6c4910b4d9788cfc81d4aca08c2 P f52f867273704793b3a3b4aca920afcf: Deleting consensus metadata
I20260924 15:28:03.199756 12924 raft_consensus.cc:2258] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:03.200496 12926 ts_tablet_manager.cc:1905] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:03.200922 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.192808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.5.98.252:34435)
0924 15:28:03.192851 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.192853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.193470 (+ 617us) server_negotiation.cc:407] Connection header received
0924 15:28:03.193758 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.193760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.193771 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.193803 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.194681 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.195438 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.196468 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.196683 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.196765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.198013 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.198015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.198015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.198022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.199431 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.199938 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.199940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.199940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.199998 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.200232 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.200233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.200233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.200376 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.200449 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.200748 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.200829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:28:03.202150 12485 connection.cc:535] server connection from 127.0.0.1:38690 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:28:03.202325 12926 ts_tablet_manager.cc:1918] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:03.202415 12926 log.cc:1196] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/2c90b6c4910b4d9788cfc81d4aca08c2
I20260924 15:28:03.202694 12926 ts_tablet_manager.cc:1939] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 783574c66bc04c82a98647ad9e156ba6: Deleting consensus metadata
W20260924 15:28:03.203034 12621 connection.cc:441] server connection from 127.0.0.1:38236 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38236 (request call id 1) could send its response
I20260924 15:28:03.204013 12924 ts_tablet_manager.cc:1905] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:03.205824 12924 ts_tablet_manager.cc:1918] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:03.205891 12924 log.cc:1196] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/2c90b6c4910b4d9788cfc81d4aca08c2
I20260924 15:28:03.206138 12924 ts_tablet_manager.cc:1939] T 2c90b6c4910b4d9788cfc81d4aca08c2 P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting consensus metadata
W20260924 15:28:03.206692 12759 connection.cc:441] server connection from 127.0.0.1:33898 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33898 (request call id 1) could send its response
I20260924 15:28:03.211211 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.204077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.5.98.253:44065)
0924 15:28:03.204120 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.204122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.204464 (+ 342us) server_negotiation.cc:407] Connection header received
0924 15:28:03.204560 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.204561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.204575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.204617 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.205389 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.206191 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.207135 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.207353 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.207435 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.208664 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.208665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.208665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.208673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.210051 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.210411 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.210413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.210413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.210467 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.210677 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.210678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.210678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.210792 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.210897 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.210998 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.211082 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:28:03.217187 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:03.204389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.5.98.252:34435)
0924 15:28:03.204499 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:03.204501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:03.211210 (+ 6709us) server_negotiation.cc:407] Connection header received
0924 15:28:03.211526 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:03.211528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:03.211541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:03.211572 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:03.212133 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.212604 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.213266 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:03.213472 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:03.213539 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:03.214665 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:03.214666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:03.214667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:03.214674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:03.216078 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.216435 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.216436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.216436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.216490 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:03.216688 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:03.216689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:03.216690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:03.216811 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:03.216929 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:03.217009 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:28:03.217080 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:28:04.187326 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.187023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:48057)
0924 15:28:04.187159 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.187262 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260924 15:28:04.187692 12869 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.188225 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.188006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:33303)
0924 15:28:04.188106 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.188177 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":60,"threads_started":1}
W20260924 15:28:04.188405 12733 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.189186 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.189003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:41135)
0924 15:28:04.189100 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.189154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
W20260924 15:28:04.189378 12596 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41153 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.692178 12937 raft_consensus.cc:482] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51cd0b5a18634343ab87689c36f3b53b)
I20260924 15:28:04.692335 12937 raft_consensus.cc:504] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } }
I20260924 15:28:04.692634 12937 leader_election.cc:288] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153), a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252:34435)
I20260924 15:28:04.692669 12938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 51cd0b5a18634343ab87689c36f3b53b)
I20260924 15:28:04.692799 12938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } }
I20260924 15:28:04.692899 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.692703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40560)
0924 15:28:04.692803 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.692851 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
W20260924 15:28:04.693011 12216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.693240 12938 leader_election.cc:288] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153), affd0a0787d248e89c75351151cb11ec (127.5.98.253:44065)
I20260924 15:28:04.693449 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.693134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40562)
0924 15:28:04.693274 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.693404 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":54,"threads_started":1}
I20260924 15:28:04.693480 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.693292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40578)
0924 15:28:04.693423 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.693450 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260924 15:28:04.693562 12216 leader_election.cc:334] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153): Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
W20260924 15:28:04.693782 12402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.694191 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.694090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40584)
0924 15:28:04.694137 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.694167 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:28:04.694320 12402 leader_election.cc:334] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153): Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.696008 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.692927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.5.98.252:34435)
0924 15:28:04.693020 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.693022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.693157 (+ 135us) server_negotiation.cc:407] Connection header received
0924 15:28:04.693270 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.693271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.693283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.693324 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.693550 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.694319 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695255 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.695636 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695758 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.695770 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.695908 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.695949 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260924 15:28:04.696013 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.692781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:34435 (local address 127.0.0.1:52080)
0924 15:28:04.693071 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.693080 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.693187 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.693402 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.693403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.693406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.693466 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.693468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.694388 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.694389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.695155 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.695157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.695760 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.695824 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.695828 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.695899 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.695945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":74,"threads_started":1}
I20260924 15:28:04.696225 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "affd0a0787d248e89c75351151cb11ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" is_pre_election: true
I20260924 15:28:04.696326 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.693147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44065 (local address 127.0.0.1:54534)
0924 15:28:04.693333 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.693342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.693394 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.693618 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.693619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.693621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.693674 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.693676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.694592 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.694593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.695345 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.695347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.695994 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.696165 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.696170 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.696217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.696263 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":38,"threads_started":1}
I20260924 15:28:04.696350 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.693317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.5.98.253:44065)
0924 15:28:04.693488 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.693490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.693497 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:04.693520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.693521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.693533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.693737 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.693771 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.694513 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695448 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.695900 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.695990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.696002 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.696238 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.696280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260924 15:28:04.696329 12427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate affd0a0787d248e89c75351151cb11ec in term 1.
I20260924 15:28:04.696514 12218 leader_election.cc:302] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a91dafb5c3c34f07b9da5b55b2af386e, affd0a0787d248e89c75351151cb11ec; no voters: 51cd0b5a18634343ab87689c36f3b53b
I20260924 15:28:04.696611 12937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 15:28:04.696599 12241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "affd0a0787d248e89c75351151cb11ec" is_pre_election: true
I20260924 15:28:04.696686 12937 raft_consensus.cc:482] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Starting leader election (detected failure of leader 51cd0b5a18634343ab87689c36f3b53b)
I20260924 15:28:04.696712 12937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:28:04.697386 12937 raft_consensus.cc:504] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } }
I20260924 15:28:04.697531 12937 leader_election.cc:288] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 election: Requested vote from peers 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153), a91dafb5c3c34f07b9da5b55b2af386e (127.5.98.252:34435)
I20260924 15:28:04.697566 12241 raft_consensus.cc:2379] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a91dafb5c3c34f07b9da5b55b2af386e in current term 2: Already voted for candidate affd0a0787d248e89c75351151cb11ec in this term.
I20260924 15:28:04.697775 12404 leader_election.cc:302] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91dafb5c3c34f07b9da5b55b2af386e; no voters: 51cd0b5a18634343ab87689c36f3b53b, affd0a0787d248e89c75351151cb11ec
I20260924 15:28:04.697837 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.697658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40600)
0924 15:28:04.697779 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.697813 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 15:28:04.697901 12938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:28:04.697973 12427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "affd0a0787d248e89c75351151cb11ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a91dafb5c3c34f07b9da5b55b2af386e"
I20260924 15:28:04.698122 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.698049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40602)
0924 15:28:04.698082 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.698101 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 15:28:04.698177 12216 leader_election.cc:334] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153): Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
I20260924 15:28:04.698622 12938 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:28:04.699224 12427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate affd0a0787d248e89c75351151cb11ec in term 2.
I20260924 15:28:04.699401 12218 leader_election.cc:302] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a91dafb5c3c34f07b9da5b55b2af386e, affd0a0787d248e89c75351151cb11ec; no voters: 51cd0b5a18634343ab87689c36f3b53b
I20260924 15:28:04.699487 12937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:28:04.699623 12937 raft_consensus.cc:686] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [term 2 LEADER]: Becoming Leader. State: Replica: affd0a0787d248e89c75351151cb11ec, State: Running, Role: LEADER
I20260924 15:28:04.699716 12937 consensus_queue.cc:237] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } }
I20260924 15:28:04.700062 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader affd0a0787d248e89c75351151cb11ec. Latest consensus state: current_term: 2 leader_uuid: "affd0a0787d248e89c75351151cb11ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:04.700155 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:04.700320 12949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:04.700758 12949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:04.700826 12949 catalog_manager.cc:1259] Loaded cluster ID: 661240a6ed4a431e8088fd83cee4e97a
I20260924 15:28:04.700865 12949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:28:04.701021 12949 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:04.701084 12949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Loaded TSK: 0
I20260924 15:28:04.701242 12949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:28:04.751508 12230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:04.753157 12230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:04.754207 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.754092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40626)
0924 15:28:04.754140 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.754176 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:28:04.754238 12427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [term 2 FOLLOWER]: Refusing update from remote peer affd0a0787d248e89c75351151cb11ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 15:28:04.754616 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.754499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.0.0.1:40634)
0924 15:28:04.754544 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.754567 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 15:28:04.754558 12947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 15:28:04.754688 12216 consensus_peers.cc:487] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec -> Peer 51cd0b5a18634343ab87689c36f3b53b (127.5.98.254:41153): Couldn't send request to peer 51cd0b5a18634343ab87689c36f3b53b. Status: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:28:04.755132 12938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: New leader affd0a0787d248e89c75351151cb11ec. Latest consensus state: current_term: 2 leader_uuid: "affd0a0787d248e89c75351151cb11ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:04.755223 12938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:28:04.755589 12950 mvcc.cc:205] Tried to move back new op lower bound from 7332920052751081472 to 7332920052530696192. Current Snapshot: MvccSnapshot[applied={T|T < 7332920052751081472}]
I20260924 15:28:04.755733 12937 sys_catalog.cc:455] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "affd0a0787d248e89c75351151cb11ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:04.755824 12937 sys_catalog.cc:458] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:04.755880 12938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "affd0a0787d248e89c75351151cb11ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "51cd0b5a18634343ab87689c36f3b53b" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41153 } } peers { permanent_uuid: "affd0a0787d248e89c75351151cb11ec" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44065 } attrs { promote: false } } peers { permanent_uuid: "a91dafb5c3c34f07b9da5b55b2af386e" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 34435 } attrs { promote: false } } }
I20260924 15:28:04.755980 12938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91dafb5c3c34f07b9da5b55b2af386e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:28:04.756176 12951 mvcc.cc:205] Tried to move back new op lower bound from 7332920052751081472 to 7332920052530696192. Current Snapshot: MvccSnapshot[applied={T|T < 7332920052751081472}]
I20260924 15:28:04.760814 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:39463 (local address 127.0.0.1:33908)
0924 15:28:04.757585 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.757595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.757642 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.758042 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.758043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.758045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.758107 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.758108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.759055 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.759056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.759879 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.759881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760569 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.760647 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.760651 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.760708 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.760746 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 15:28:04.760851 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33908 (local address 127.5.98.195:39463)
0924 15:28:04.757893 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.757896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.757904 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:04.757929 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.757929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.757940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.758034 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.758195 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.758946 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.759952 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760488 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760562 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.760573 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.760722 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.760768 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":56,"threads_started":1}
I20260924 15:28:04.761045 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:37111 (local address 127.0.0.1:38256)
0924 15:28:04.757548 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.757559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.757597 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.757967 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.757968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.757970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.758022 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.758024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.759021 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.759022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.760102 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.760104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760831 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.760894 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.760900 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.760934 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.760977 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:28:04.761080 12803 tablet_service.cc:1467] Processing CreateTablet for tablet c04d384b0e0e421e9a5913eba705078a (DEFAULT_TABLE table=test-table [id=4ee895995e6347b1af8297cc1d793526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:04.761160 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:45047 (local address 127.0.0.1:38702)
0924 15:28:04.757760 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.757770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.757825 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.758101 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.758102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.758104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.758143 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.758144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.758816 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.758817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.760214 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.760216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760935 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.760999 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.761002 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.761049 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.761088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":188,"thread_start_us":74,"threads_started":1}
I20260924 15:28:04.761209 12803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04d384b0e0e421e9a5913eba705078a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:04.761366 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.5.98.193:45047)
0924 15:28:04.757784 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.757787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.757811 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:28:04.757987 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.757988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.757998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.758058 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.758218 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.758711 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760299 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760823 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760887 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.760902 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.761239 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.761285 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260924 15:28:04.761586 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.757561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.5.98.194:37111)
0924 15:28:04.757835 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.757838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.757846 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:04.757874 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.757875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.757888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.757954 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.758203 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.758946 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.760173 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.760740 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.761373 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.761389 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.761408 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.761515 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":72,"threads_started":1}
I20260924 15:28:04.762651 12959 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Bootstrap starting.
I20260924 15:28:04.762835 12530 tablet_service.cc:1467] Processing CreateTablet for tablet c04d384b0e0e421e9a5913eba705078a (DEFAULT_TABLE table=test-table [id=4ee895995e6347b1af8297cc1d793526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:04.762854 12667 tablet_service.cc:1467] Processing CreateTablet for tablet c04d384b0e0e421e9a5913eba705078a (DEFAULT_TABLE table=test-table [id=4ee895995e6347b1af8297cc1d793526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:04.762965 12530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04d384b0e0e421e9a5913eba705078a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:04.762993 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04d384b0e0e421e9a5913eba705078a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:04.763121 12959 tablet_bootstrap.cc:654] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:04.763845 12959 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: No bootstrap required, opened a new log
I20260924 15:28:04.763921 12959 ts_tablet_manager.cc:1397] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:04.764066 12959 raft_consensus.cc:348] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.764149 12959 raft_consensus.cc:374] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:04.764173 12959 raft_consensus.cc:729] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9269a6f59fe94cfc99757bc2276b1a2d, State: Initialized, Role: FOLLOWER
I20260924 15:28:04.764254 12959 consensus_queue.cc:260] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.764434 12959 ts_tablet_manager.cc:1428] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:04.764643 12961 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: Bootstrap starting.
I20260924 15:28:04.765050 12961 tablet_bootstrap.cc:654] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:04.765067 12871 heartbeater.cc:503] Master 127.5.98.253:44065 was elected leader, sending a full tablet report...
I20260924 15:28:04.765326 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.764559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:57243)
0924 15:28:04.765254 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.765291 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":82,"threads_started":1}
I20260924 15:28:04.765703 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.765535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:46013)
0924 15:28:04.765642 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.765676 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":75,"threads_started":1}
I20260924 15:28:04.765695 12963 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Bootstrap starting.
I20260924 15:28:04.765952 12961 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: No bootstrap required, opened a new log
I20260924 15:28:04.766021 12961 ts_tablet_manager.cc:1397] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:04.766101 12963 tablet_bootstrap.cc:654] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:04.766167 12961 raft_consensus.cc:348] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.766258 12961 raft_consensus.cc:374] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:04.766326 12961 raft_consensus.cc:729] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52f867273704793b3a3b4aca920afcf, State: Initialized, Role: FOLLOWER
I20260924 15:28:04.766387 12961 consensus_queue.cc:260] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.766566 12961 ts_tablet_manager.cc:1428] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:04.766899 12963 tablet_bootstrap.cc:492] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: No bootstrap required, opened a new log
I20260924 15:28:04.767041 12963 ts_tablet_manager.cc:1397] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:04.767050 12598 heartbeater.cc:503] Master 127.5.98.253:44065 was elected leader, sending a full tablet report...
I20260924 15:28:04.767112 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.766683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:51125)
0924 15:28:04.767041 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.767081 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":97,"threads_started":1}
I20260924 15:28:04.767174 12963 raft_consensus.cc:348] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.767254 12963 raft_consensus.cc:374] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:04.767277 12963 raft_consensus.cc:729] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783574c66bc04c82a98647ad9e156ba6, State: Initialized, Role: FOLLOWER
I20260924 15:28:04.767382 12963 consensus_queue.cc:260] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.767486 12962 raft_consensus.cc:482] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:04.767531 12963 ts_tablet_manager.cc:1428] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:04.767552 12962 raft_consensus.cc:504] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.767961 12962 leader_election.cc:288] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111), f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
I20260924 15:28:04.768095 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.767757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:58345)
0924 15:28:04.767966 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.768006 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260924 15:28:04.768321 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.767811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:52065)
0924 15:28:04.768221 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.768288 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
I20260924 15:28:04.768815 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.768729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:41591)
0924 15:28:04.768767 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.768790 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 15:28:04.769032 12735 heartbeater.cc:503] Master 127.5.98.253:44065 was elected leader, sending a full tablet report...
I20260924 15:28:04.771303 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.768044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:37111 (local address 127.5.98.195:45657)
0924 15:28:04.768135 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.768146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.768190 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.768470 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.768471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.768474 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.768572 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.768574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.769481 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.769482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.770451 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.770454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.771044 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.771046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.771127 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.771136 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.771196 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.771231 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 15:28:04.771361 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.768257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:45657 (local address 127.5.98.194:37111)
0924 15:28:04.768315 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.768317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.768326 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:04.768351 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.768351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.768377 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.768445 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.768698 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.769402 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.770530 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.770970 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.771029 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.771036 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.771244 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.771309 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 15:28:04.771399 12599 tablet.cc:2369] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:04.771791 12687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04d384b0e0e421e9a5913eba705078a" candidate_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783574c66bc04c82a98647ad9e156ba6" is_pre_election: true
I20260924 15:28:04.771906 12687 raft_consensus.cc:2454] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9269a6f59fe94cfc99757bc2276b1a2d in term 0.
I20260924 15:28:04.772102 12756 leader_election.cc:302] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783574c66bc04c82a98647ad9e156ba6, 9269a6f59fe94cfc99757bc2276b1a2d; no voters:
I20260924 15:28:04.772217 12962 raft_consensus.cc:2781] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:04.772255 12962 raft_consensus.cc:482] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:04.772275 12962 raft_consensus.cc:3037] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:04.772477 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.768321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:32781 (local address 127.5.98.193:45047)
0924 15:28:04.768352 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.768353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.769847 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 15:28:04.769873 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.769873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.769883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.769938 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:04.770084 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.770617 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.771690 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.772178 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.772257 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.772265 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.772412 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.772440 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:28:04.772534 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.768171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:45047 (local address 127.5.98.195:32781)
0924 15:28:04.768452 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.768462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.768510 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.769946 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:04.769947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:04.769949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:04.769996 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.769998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.770700 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.770701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.771581 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:04.771583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.772257 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.772259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:04.772336 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.772341 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:28:04.772399 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:28:04.772439 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260924 15:28:04.772748 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04d384b0e0e421e9a5913eba705078a" candidate_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52f867273704793b3a3b4aca920afcf" is_pre_election: true
I20260924 15:28:04.772835 12550 raft_consensus.cc:2454] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9269a6f59fe94cfc99757bc2276b1a2d in term 0.
I20260924 15:28:04.773068 12962 raft_consensus.cc:504] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.773205 12962 leader_election.cc:288] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [CANDIDATE]: Term 1 election: Requested vote from peers 783574c66bc04c82a98647ad9e156ba6 (127.5.98.194:37111), f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047)
I20260924 15:28:04.773320 12687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04d384b0e0e421e9a5913eba705078a" candidate_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "783574c66bc04c82a98647ad9e156ba6"
I20260924 15:28:04.773322 12550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04d384b0e0e421e9a5913eba705078a" candidate_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52f867273704793b3a3b4aca920afcf"
I20260924 15:28:04.773387 12550 raft_consensus.cc:3037] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:04.773403 12687 raft_consensus.cc:3037] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:04.774010 12687 raft_consensus.cc:2454] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269a6f59fe94cfc99757bc2276b1a2d in term 1.
I20260924 15:28:04.774019 12550 raft_consensus.cc:2454] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9269a6f59fe94cfc99757bc2276b1a2d in term 1.
I20260924 15:28:04.774210 12756 leader_election.cc:302] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783574c66bc04c82a98647ad9e156ba6, 9269a6f59fe94cfc99757bc2276b1a2d; no voters:
I20260924 15:28:04.774323 12962 raft_consensus.cc:2781] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:04.774448 12962 raft_consensus.cc:686] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 LEADER]: Becoming Leader. State: Replica: 9269a6f59fe94cfc99757bc2276b1a2d, State: Running, Role: LEADER
I20260924 15:28:04.774521 12962 consensus_queue.cc:237] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } }
I20260924 15:28:04.774700 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.774594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:47859)
0924 15:28:04.774649 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.774680 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:28:04.775249 12230 catalog_manager.cc:5495] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195). New cstate: current_term: 1 leader_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9269a6f59fe94cfc99757bc2276b1a2d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 39463 } health_report { overall_health: HEALTHY } } }
I20260924 15:28:04.782863 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.780899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.5.98.195:39463)
0924 15:28:04.780950 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:04.780952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:04.780961 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:04.781026 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:04.781027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:04.781042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:04.781092 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:04.781199 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.781622 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.782270 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:04.782440 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:04.782515 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:04.782634 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:04.782723 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:04.782774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:04.782807 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 15:28:04.782822 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:28:04.783869 12687 raft_consensus.cc:1261] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Refusing update from remote peer 9269a6f59fe94cfc99757bc2276b1a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:04.784083 12550 raft_consensus.cc:1261] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Refusing update from remote peer 9269a6f59fe94cfc99757bc2276b1a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:04.784170 12962 consensus_queue.cc:1035] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "783574c66bc04c82a98647ad9e156ba6" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:04.784286 12972 consensus_queue.cc:1035] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52f867273704793b3a3b4aca920afcf" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:04.784895 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.784807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:53317)
0924 15:28:04.784839 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.784863 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 15:28:04.785173 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.785085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:45759)
0924 15:28:04.785118 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.785146 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 15:28:04.785183 12872 tablet.cc:2369] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:04.786425 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.786325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.195:34601)
0924 15:28:04.786360 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.786385 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 15:28:04.786428 12977 mvcc.cc:205] Tried to move back new op lower bound from 7332920052872679424 to 7332920052837036032. Current Snapshot: MvccSnapshot[applied={T|T < 7332920052872679424}]
I20260924 15:28:04.786665 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.786578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.194:57877)
0924 15:28:04.786613 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.786637 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:28:04.786782 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:04.786698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41153 (local address 127.5.98.193:44177)
0924 15:28:04.786736 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.786758 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:41153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:28:04.788522 12230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:04.788604 12230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:04.789783 12230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P affd0a0787d248e89c75351151cb11ec: Sending DeleteTablet for 3 replicas of tablet c04d384b0e0e421e9a5913eba705078a
I20260924 15:28:04.790088 12667 tablet_service.cc:1514] Processing DeleteTablet for tablet c04d384b0e0e421e9a5913eba705078a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:04 UTC) from {username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:38256
I20260924 15:28:04.790107 12803 tablet_service.cc:1514] Processing DeleteTablet for tablet c04d384b0e0e421e9a5913eba705078a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:04 UTC) from {username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:33908
I20260924 15:28:04.790285 12985 tablet_replica.cc:330] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: stopping tablet replica
I20260924 15:28:04.790361 12985 raft_consensus.cc:2229] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:04.790393 12530 tablet_service.cc:1514] Processing DeleteTablet for tablet c04d384b0e0e421e9a5913eba705078a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:04 UTC) from {username='oryx', principal='oryx/127.5.98.253@KRBTEST.COM'} at 127.0.0.1:38702
I20260924 15:28:04.790477 12985 raft_consensus.cc:2258] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:04.790594 12986 tablet_replica.cc:330] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: stopping tablet replica
I20260924 15:28:04.790603 12987 tablet_replica.cc:330] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf: stopping tablet replica
I20260924 15:28:04.790648 12987 raft_consensus.cc:2229] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:04.790676 12986 raft_consensus.cc:2229] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:04.790700 12987 raft_consensus.cc:2258] T c04d384b0e0e421e9a5913eba705078a P f52f867273704793b3a3b4aca920afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:04.790752 12986 raft_consensus.cc:2258] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:04.790880 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 12378
I20260924 15:28:04.791164 12986 ts_tablet_manager.cc:1905] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:04.793426 12985 ts_tablet_manager.cc:1905] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:04.794979 12985 ts_tablet_manager.cc:1918] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:04.795077 12985 log.cc:1196] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/c04d384b0e0e421e9a5913eba705078a
I20260924 15:28:04.795368 12985 ts_tablet_manager.cc:1939] T c04d384b0e0e421e9a5913eba705078a P 9269a6f59fe94cfc99757bc2276b1a2d: Deleting consensus metadata
I20260924 15:28:04.795744 12216 catalog_manager.cc:4841] TS 9269a6f59fe94cfc99757bc2276b1a2d (127.5.98.195:39463): tablet c04d384b0e0e421e9a5913eba705078a (table test-table [id=4ee895995e6347b1af8297cc1d793526]) successfully deleted
W20260924 15:28:04.798198 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:04.798048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:45047 (local address 127.0.0.1:38714)
0924 15:28:04.798108 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.798144 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:45047: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:28:04.798374 12218 catalog_manager.cc:4584] TS f52f867273704793b3a3b4aca920afcf (127.5.98.193:45047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c04d384b0e0e421e9a5913eba705078a: Network error: Client connection negotiation failed: client connection to 127.5.98.193:45047: connect: Connection reset by peer (error 104)
I20260924 15:28:04.799782 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 12604
I20260924 15:28:04.799835 12986 ts_tablet_manager.cc:1918] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:04.799894 12986 log.cc:1196] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/c04d384b0e0e421e9a5913eba705078a
I20260924 15:28:04.800187 12986 ts_tablet_manager.cc:1939] T c04d384b0e0e421e9a5913eba705078a P 783574c66bc04c82a98647ad9e156ba6: Deleting consensus metadata
W20260924 15:28:04.806713 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:04.806478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:37111 (local address 127.0.0.1:38266)
0924 15:28:04.806522 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:04.806532 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:04.806572 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:04.806663 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.194:37111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:28:04.806911 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 12740
I20260924 15:28:04.812084 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 12097
I20260924 15:28:04.816893 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 12285
2026-09-24T15:28:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4521 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:04 dist-test-slave-4g6f krb5kdc[12990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:04 dist-test-slave-4g6f krb5kdc[12990](info): set up 2 sockets
Sep 24 15:28:04 dist-test-slave-4g6f krb5kdc[12990](info): commencing operation
krb5kdc: starting...
W20260924 15:28:05.877017 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:05 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab.
Entry for principal oryx/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab.
I20260924 15:28:05.994805 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:46827
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab
--principal=oryx/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:06.076160 13003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:06.076402 13003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:06.076440 13003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:06.077585 13003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:06.077646 13003 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:28:06.077667 13003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:06.077689 13003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:06.077718 13003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:06.077739 13003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:06.079242 13003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:46827
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.254.keytab
--principal=oryx/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:06.079520 13003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:06.079753 13003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:06.084441 13009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.084522 13008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.084388 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:06.084705 13003 server_base.cc:1034] running on GCE node
I20260924 15:28:06.084936 13003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:06.085165 13003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:06.086306 13003 hybrid_clock.cc:648] HybridClock initialized: now 1790263686086298 us; error 32 us; skew 500 ppm
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:06.088963 13003 init.cc:377] Logged in from keytab as oryx/127.5.98.254@KRBTEST.COM (short username oryx)
I20260924 15:28:06.089387 13003 webserver.cc:460] Webserver started at http://127.5.98.254:46489/ using document root <none> and password file <none>
I20260924 15:28:06.089633 13003 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:06.089701 13003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:06.089844 13003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:06.090857 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "f4fa2b99fda5428ca1d981f9c4597cf6"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "83049ca5ed986652be4e9bed12e80085"
server_key_iv: "aa13b6737d5d08810330c7ed2a33b5ac"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.091223 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "f4fa2b99fda5428ca1d981f9c4597cf6"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "83049ca5ed986652be4e9bed12e80085"
server_key_iv: "aa13b6737d5d08810330c7ed2a33b5ac"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.092619 13003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:06.093334 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.093515 13003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:06.093622 13003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "f4fa2b99fda5428ca1d981f9c4597cf6"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "83049ca5ed986652be4e9bed12e80085"
server_key_iv: "aa13b6737d5d08810330c7ed2a33b5ac"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.093694 13003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:06.104943 13003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:06.107205 13003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:06.107364 13003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:06.112010 13003 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:46827
I20260924 15:28:06.112056 13080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:46827 every 8 connection(s)
I20260924 15:28:06.112385 13003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:06.112989 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.115458 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Bootstrap starting.
I20260924 15:28:06.116112 13081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.116420 13081 log.cc:824] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:06.117120 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: No bootstrap required, opened a new log
I20260924 15:28:06.119359 13081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } }
I20260924 15:28:06.119572 13081 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.119572 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13003
I20260924 15:28:06.119617 13081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4fa2b99fda5428ca1d981f9c4597cf6, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.119702 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:06.119823 13081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } }
I20260924 15:28:06.119920 13081 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:06.119923 5515 external_mini_cluster.cc:1428] Setting key a92eb68fc7b24c789464b1c738c22aaf
I20260924 15:28:06.119958 13081 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:06.120018 13081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:06.120805 13081 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } }
I20260924 15:28:06.120921 13081 leader_election.cc:302] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4fa2b99fda5428ca1d981f9c4597cf6; no voters:
I20260924 15:28:06.121115 13081 leader_election.cc:288] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:06.121165 13086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:06.121336 13086 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [term 1 LEADER]: Becoming Leader. State: Replica: f4fa2b99fda5428ca1d981f9c4597cf6, State: Running, Role: LEADER
I20260924 15:28:06.121385 13081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:06.121531 13086 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } }
I20260924 15:28:06.121997 13087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } } }
I20260924 15:28:06.122088 13087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:06.122573 13094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:06.123063 13094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:06.123175 13088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4fa2b99fda5428ca1d981f9c4597cf6. Latest consensus state: current_term: 1 leader_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fa2b99fda5428ca1d981f9c4597cf6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46827 } } }
I20260924 15:28:06.123641 13088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:06.124770 13094 catalog_manager.cc:1347] Generated new cluster ID: 0f2e66377ceb4d8190275fdf56e0668c
I20260924 15:28:06.124826 13094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:06.132151 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.120402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.5.98.254:46827)
0924 15:28:06.120631 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.120643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.120681 (+ 38us) server_negotiation.cc:407] Connection header received
0924 15:28:06.120807 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.120815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.120878 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.121045 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.121832 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.123536 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.124660 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.125313 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.125434 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.128323 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.128327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.128327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.128339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.130416 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.130832 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.130835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.130836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.130953 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.131164 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.131165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.131166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.131494 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.131685 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.131791 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.131937 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260924 15:28:06.135389 13094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:06.136014 13094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:06.138823 13094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Generated new TSK 0
I20260924 15:28:06.139026 13094 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
I20260924 15:28:06.166025 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:46827
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:06.248093 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:06.248298 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:06.248322 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:06.249445 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:06.249480 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:06.249500 13107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:28:06.249518 13107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:06.251003 13107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:06.251185 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:06.251360 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:28:06.256240 13107 server_base.cc:1034] running on GCE node
W20260924 15:28:06.256273 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.256358 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.256358 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:06.256690 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:06.256927 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:06.258080 13107 hybrid_clock.cc:648] HybridClock initialized: now 1790263686258071 us; error 45 us; skew 500 ppm
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:06.260677 13107 init.cc:377] Logged in from keytab as oryx/127.5.98.193@KRBTEST.COM (short username oryx)
I20260924 15:28:06.261122 13107 webserver.cc:460] Webserver started at http://127.5.98.193:45721/ using document root <none> and password file <none>
I20260924 15:28:06.261355 13107 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:06.261427 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:06.261562 13107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:06.262563 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "774a314c131a46abb4e0f5aaeea6dab7"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "452317551b4a74f494eedbeecee898f6"
server_key_iv: "6457c34aac48efa790e609d8a974f337"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.262920 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "774a314c131a46abb4e0f5aaeea6dab7"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "452317551b4a74f494eedbeecee898f6"
server_key_iv: "6457c34aac48efa790e609d8a974f337"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.264273 13107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:28:06.264981 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.265141 13107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:06.265228 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "774a314c131a46abb4e0f5aaeea6dab7"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "452317551b4a74f494eedbeecee898f6"
server_key_iv: "6457c34aac48efa790e609d8a974f337"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.265302 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:06.285712 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:06.288072 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:06.288225 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:06.288758 13107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:06.288820 13107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.288879 13107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:06.288904 13107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:06.296025 13107 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:44063
I20260924 15:28:06.296131 13237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:44063 every 8 connection(s)
I20260924 15:28:06.296460 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:06.298800 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.289302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:38747 (local address 127.5.98.254:46827)
0924 15:28:06.289345 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.289347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.289609 (+ 262us) server_negotiation.cc:407] Connection header received
0924 15:28:06.290110 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.290112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.290129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.290184 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.291070 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.291873 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.292874 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.293098 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.293184 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.296148 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.296150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.296150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.296162 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.297685 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.298066 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.298067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.298067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.298131 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.298302 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.298303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.298303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.298407 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.298456 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.298672 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.298738 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:06.298964 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.289303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.193:38747)
0924 15:28:06.289474 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.289509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.290004 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.290267 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.290270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.290668 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.290973 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.290981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.291968 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.291969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.292783 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.292785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.293206 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.293207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.293348 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.293866 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.293876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.296020 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.297756 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.297763 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.297765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.297991 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.298170 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.298170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.298171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.298255 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.298448 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.298452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.298566 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.298648 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.298767 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260924 15:28:06.299475 13238 heartbeater.cc:347] Connected to a master server at 127.5.98.254:46827
I20260924 15:28:06.299558 13238 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:06.299775 13238 heartbeater.cc:511] Master 127.5.98.254:46827 requested a full tablet report, sending...
I20260924 15:28:06.300396 13032 ts_manager.cc:194] Registered new tserver with Master: 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063)
I20260924 15:28:06.300614 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13107
I20260924 15:28:06.300729 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:06.300915 5515 external_mini_cluster.cc:1428] Setting key 6f093d7f31605edebec4f1c4e4c2b2dc
I20260924 15:28:06.301350 13032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:38747
I20260924 15:28:06.306358 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.300552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:37223 (local address 127.5.98.254:46827)
0924 15:28:06.300589 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.300591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.300812 (+ 221us) server_negotiation.cc:407] Connection header received
0924 15:28:06.300912 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.300913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.300922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.300948 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.301333 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.301854 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.302617 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.302931 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.303009 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.303942 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.303944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.303944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.303951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.305265 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.305609 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.305610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.305610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.305664 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.305861 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.305862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.305862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.305975 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.306041 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.306203 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.306274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:28:06.306362 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.300620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.193:37223)
0924 15:28:06.300753 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.300763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.300835 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.301000 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.301001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.301188 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.301251 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.301253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.301938 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.301939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.302527 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.302529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.303025 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.303026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.303083 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.303499 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.303504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.303836 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.305364 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.305365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.305366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.305539 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.305741 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.305742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.305742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.305802 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.306045 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.306046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.306092 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.306163 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.306267 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
I20260924 15:28:06.324489 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:46827
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:06.404294 13247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:06.404493 13247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:06.404515 13247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:06.405668 13247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:06.405704 13247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:06.405725 13247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:28:06.405786 13247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:06.407222 13247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:06.407454 13247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:06.407672 13247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:06.412443 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.412494 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.412457 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:06.412863 13247 server_base.cc:1034] running on GCE node
I20260924 15:28:06.413057 13247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:06.413239 13247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:06.414382 13247 hybrid_clock.cc:648] HybridClock initialized: now 1790263686414361 us; error 34 us; skew 500 ppm
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:06.416821 13247 init.cc:377] Logged in from keytab as oryx/127.5.98.194@KRBTEST.COM (short username oryx)
I20260924 15:28:06.417227 13247 webserver.cc:460] Webserver started at http://127.5.98.194:36653/ using document root <none> and password file <none>
I20260924 15:28:06.417454 13247 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:06.417500 13247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:06.417613 13247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:06.418603 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "1cd587e7b09bc7199f3138e2a215864f"
server_key_iv: "852c7dcc5d0d961808e4ee3cd9437014"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.418938 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "1cd587e7b09bc7199f3138e2a215864f"
server_key_iv: "852c7dcc5d0d961808e4ee3cd9437014"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.420291 13247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 15:28:06.421000 13262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.421167 13247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:06.421267 13247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "1cd587e7b09bc7199f3138e2a215864f"
server_key_iv: "852c7dcc5d0d961808e4ee3cd9437014"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.421335 13247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:06.467079 13247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:06.469202 13247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:06.469365 13247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:06.469887 13247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:06.469949 13247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.469995 13247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:06.470021 13247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:06.477212 13247 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:33021
I20260924 15:28:06.477288 13377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:33021 every 8 connection(s)
I20260924 15:28:06.477610 13247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:28:06.479140 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13247
I20260924 15:28:06.479272 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:06.479506 5515 external_mini_cluster.cc:1428] Setting key 36ffadcd9ab1ed33b51b12c8883fac65
I20260924 15:28:06.480517 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.470480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:40203 (local address 127.5.98.254:46827)
0924 15:28:06.470538 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.470540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.470743 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:28:06.471315 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.471317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.471333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.471414 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.472285 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.472829 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.474156 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.474409 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.474504 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.477459 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.477461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.477462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.477471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.478905 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.479485 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.479486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.479487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.479542 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.479850 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.479851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.479852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.479974 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.480050 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.480344 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.480427 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:28:06.480705 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.470460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.194:40203)
0924 15:28:06.470627 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.470665 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.471181 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.471506 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.471511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.471966 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.472176 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.472181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.472934 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.472936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.474055 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.474058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.474527 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.474528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.474679 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.475277 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.475287 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.477357 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.478989 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.478997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.479000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.479408 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.479623 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.479624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.479624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.479788 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.480042 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.480048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.480206 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.480307 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.480458 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260924 15:28:06.481884 13378 heartbeater.cc:347] Connected to a master server at 127.5.98.254:46827
I20260924 15:28:06.481995 13378 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:06.482188 13378 heartbeater.cc:511] Master 127.5.98.254:46827 requested a full tablet report, sending...
I20260924 15:28:06.482587 13032 ts_manager.cc:194] Registered new tserver with Master: f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021)
I20260924 15:28:06.483146 13032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:40203
I20260924 15:28:06.488242 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.482562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:55783 (local address 127.5.98.254:46827)
0924 15:28:06.482615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.482617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.482792 (+ 175us) server_negotiation.cc:407] Connection header received
0924 15:28:06.482854 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.482855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.482866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.482926 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.483290 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.483868 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.484543 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.484733 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.484800 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.485652 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.485654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.485654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.485661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.487031 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.487425 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.487427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.487427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.487486 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.487697 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.487698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.487698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.487853 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.487926 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.488083 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.488155 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:06.488250 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.482524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.194:55783)
0924 15:28:06.482718 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.482729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.482792 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.482930 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.482930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.483125 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.483196 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.483198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.483960 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.483961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.484464 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.484465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.484812 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.484813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.484881 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.485308 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.485312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.485560 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.487138 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.487139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.487140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.487319 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.487556 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.487556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.487557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.487617 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.487933 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.487933 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.487978 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.488058 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.488157 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
I20260924 15:28:06.504619 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:46827
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:06.585865 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:06.586055 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:06.586078 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:06.587157 13387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:06.587193 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:06.587211 13387 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:28:06.587229 13387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:06.589056 13387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.98.254:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:06.589243 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:06.589419 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:06.594296 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:06.594282 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:06.594332 13387 server_base.cc:1034] running on GCE node
W20260924 15:28:06.594282 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:06.594696 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:06.594897 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:06.596061 13387 hybrid_clock.cc:648] HybridClock initialized: now 1790263686596043 us; error 47 us; skew 500 ppm
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:06.598762 13387 init.cc:377] Logged in from keytab as oryx/127.5.98.195@KRBTEST.COM (short username oryx)
I20260924 15:28:06.599202 13387 webserver.cc:460] Webserver started at http://127.5.98.195:38105/ using document root <none> and password file <none>
I20260924 15:28:06.599444 13387 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:06.599514 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:06.603775 13387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:06.605136 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "282450cf060e43ceaa8213f0a56fafba"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "432ab5ca1a100beaf6ebbfb79bac3059"
server_key_iv: "0ab17196d517cf0be928b57800f2ef99"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.605553 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "282450cf060e43ceaa8213f0a56fafba"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "432ab5ca1a100beaf6ebbfb79bac3059"
server_key_iv: "0ab17196d517cf0be928b57800f2ef99"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.607259 13387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:28:06.608209 13402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.608390 13387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:28:06.608502 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "282450cf060e43ceaa8213f0a56fafba"
format_stamp: "Formatted at 2026-09-24 15:28:06 on dist-test-slave-4g6f"
server_key: "432ab5ca1a100beaf6ebbfb79bac3059"
server_key_iv: "0ab17196d517cf0be928b57800f2ef99"
server_key_version: "encryptionkey@0"
I20260924 15:28:06.608577 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:06.625495 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:06.627816 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:06.627975 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:06.628489 13387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:06.628548 13387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:06.628589 13387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:06.628614 13387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:06.635638 13387 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:33135
I20260924 15:28:06.635707 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:33135 every 8 connection(s)
I20260924 15:28:06.636029 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:06.638758 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.628947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:56199 (local address 127.5.98.254:46827)
0924 15:28:06.628996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.628998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.629182 (+ 184us) server_negotiation.cc:407] Connection header received
0924 15:28:06.629725 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.629726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.629740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.629796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.630739 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.631549 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.633269 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.633491 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.633575 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.635859 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.635861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.635861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.635870 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.637341 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.637774 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.637775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.637776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.637837 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.638085 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.638086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.638087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.638202 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.638275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.638542 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.638668 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:28:06.638902 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.628926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.195:56199)
0924 15:28:06.629066 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.629100 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.629616 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.629919 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.629924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.630318 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.630636 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.630643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.631658 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.631659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.633175 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.633179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.633584 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.633585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.633716 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.634236 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.634244 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.635776 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.637425 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.637432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.637434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.637702 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.637911 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.637912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.637912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.638021 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.638266 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.638272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.638414 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.638518 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.638677 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260924 15:28:06.639163 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13387
I20260924 15:28:06.639263 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:06.639400 13518 heartbeater.cc:347] Connected to a master server at 127.5.98.254:46827
I20260924 15:28:06.639451 5515 external_mini_cluster.cc:1428] Setting key 69009fe0303a21c0dcc1959db1861a73
I20260924 15:28:06.639513 13518 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:06.639734 13518 heartbeater.cc:511] Master 127.5.98.254:46827 requested a full tablet report, sending...
I20260924 15:28:06.640122 13035 ts_manager.cc:194] Registered new tserver with Master: 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:06.640619 13035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:56199
I20260924 15:28:06.645500 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.640534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:49557 (local address 127.5.98.254:46827)
0924 15:28:06.640581 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.640583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.640644 (+ 61us) server_negotiation.cc:407] Connection header received
0924 15:28:06.640818 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.640819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.640828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.640882 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.641201 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.641736 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.642358 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.642558 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.642610 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.643491 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.643492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.643493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.643499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.644663 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.644950 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.644951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.644951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.644999 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.645137 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.645137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.645138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.645219 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.645263 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.645373 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.645445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:28:06.645504 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.640470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.5.98.195:49557)
0924 15:28:06.640592 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.640603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.640657 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.640892 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.640893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.641070 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:06.641129 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.641130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.641813 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.641813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.642294 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.642295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.642619 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.642619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.642665 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.643117 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:06.643128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:06.643395 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:06.644732 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.644733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.644733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.644888 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.645051 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:06.645052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:06.645052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:06.645100 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:06.645264 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:06.645265 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:06.645298 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.645360 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.645435 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260924 15:28:06.651051 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:28:06.672225 13085 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:06.666626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.5.98.254:46827)
0924 15:28:06.666689 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.666692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.666740 (+ 48us) server_negotiation.cc:407] Connection header received
0924 15:28:06.666829 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.666830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.666840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.666892 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.667513 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.668083 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.668948 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.669150 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.669214 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.672115 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53476: BlockingRecv error: recv got EOF from 127.0.0.1:53476 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 15:28:06 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:06.679088 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.673058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.5.98.254:46827)
0924 15:28:06.673128 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.673130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.673152 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:28:06.673205 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.673206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.673214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.673260 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:06.673698 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.674150 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.674791 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.674964 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.675013 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.677032 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:06.677033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:06.677033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:06.677039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:06.678188 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.678487 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.678488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.678488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.678542 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:06.678709 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:06.678710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:06.678710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:06.678804 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:06.678881 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.678958 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.679009 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:28:06.680698 13035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:06.682412 13035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:06.689188 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.685110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.5.98.193:44063)
0924 15:28:06.685261 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.685264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.685327 (+ 63us) server_negotiation.cc:407] Connection header received
0924 15:28:06.685403 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.685405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.685421 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.685494 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:06.685695 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686753 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688043 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.688737 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688817 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.688898 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.689040 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.689082 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260924 15:28:06.689189 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.684718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:33021 (local address 127.0.0.1:42224)
0924 15:28:06.684921 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.684965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.685042 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.685692 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.685693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.685695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:06.685742 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.685743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686583 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.686584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.687897 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.687900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.688830 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.688919 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.688936 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.689017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.689090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":41,"thread_start_us":87,"threads_started":1}
I20260924 15:28:06.689189 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.684903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44063 (local address 127.0.0.1:40916)
0924 15:28:06.685276 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.685286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.685338 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.685570 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.685571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.685573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:06.685617 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.685619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686828 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.686829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.687978 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.687980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.688829 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.688924 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.688936 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.689017 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.689090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":59,"thread_start_us":60,"threads_started":1}
I20260924 15:28:06.689448 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.684922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.5.98.194:33021)
0924 15:28:06.685484 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.685489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.685498 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:06.685535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.685537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.685552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.685624 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:06.685799 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686493 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.687977 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.688737 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688817 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.688896 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.689304 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.689349 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":78,"threads_started":1}
I20260924 15:28:06.690075 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc466173c224a0ab3f52041b8040103 (DEFAULT_TABLE table=test-table [id=d90c9b84ad0b4334a2d90f5577647f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.690249 13172 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc466173c224a0ab3f52041b8040103 (DEFAULT_TABLE table=test-table [id=d90c9b84ad0b4334a2d90f5577647f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.690481 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc466173c224a0ab3f52041b8040103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.690611 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc466173c224a0ab3f52041b8040103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.690675 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.684718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:33135 (local address 127.0.0.1:49916)
0924 15:28:06.685012 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.685021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.685062 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.685314 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.685318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.685320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:06.685386 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.685387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686840 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.686841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.687859 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.687862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.690435 (+ 2573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.690436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.690510 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.690514 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.690548 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.690591 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":45,"thread_start_us":43,"threads_started":1}
I20260924 15:28:06.691087 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.684812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.5.98.195:33135)
0924 15:28:06.685131 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.685137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.685149 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:06.685192 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.685195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.685211 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.685308 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:06.685558 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.686717 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.688928 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.690316 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.690832 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.690916 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.690944 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.690989 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20260924 15:28:06.691712 13452 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc466173c224a0ab3f52041b8040103 (DEFAULT_TABLE table=test-table [id=d90c9b84ad0b4334a2d90f5577647f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.692065 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc466173c224a0ab3f52041b8040103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.694042 13546 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Bootstrap starting.
I20260924 15:28:06.694459 13547 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Bootstrap starting.
I20260924 15:28:06.694830 13546 tablet_bootstrap.cc:654] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.695065 13547 tablet_bootstrap.cc:654] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.695163 13546 log.cc:824] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:06.695358 13548 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Bootstrap starting.
I20260924 15:28:06.695431 13547 log.cc:824] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:06.696136 13546 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: No bootstrap required, opened a new log
I20260924 15:28:06.696233 13546 ts_tablet_manager.cc:1397] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:28:06.696324 13548 tablet_bootstrap.cc:654] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.696455 13547 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: No bootstrap required, opened a new log
I20260924 15:28:06.696527 13547 ts_tablet_manager.cc:1397] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:06.696699 13548 log.cc:824] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:06.697459 13548 tablet_bootstrap.cc:492] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: No bootstrap required, opened a new log
I20260924 15:28:06.697531 13548 ts_tablet_manager.cc:1397] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:28:06.698035 13546 raft_consensus.cc:348] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.698184 13546 raft_consensus.cc:374] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.698225 13546 raft_consensus.cc:729] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.698330 13546 consensus_queue.cc:260] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.698621 13546 ts_tablet_manager.cc:1428] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:28:06.698544 13547 raft_consensus.cc:348] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.698686 13547 raft_consensus.cc:374] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.698727 13547 raft_consensus.cc:729] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d4dc0c49a0406ab103bbd8995ba1da, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.698632 13238 heartbeater.cc:503] Master 127.5.98.254:46827 was elected leader, sending a full tablet report...
I20260924 15:28:06.698904 13547 consensus_queue.cc:260] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.699160 13547 ts_tablet_manager.cc:1428] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:28:06.699275 13378 heartbeater.cc:503] Master 127.5.98.254:46827 was elected leader, sending a full tablet report...
I20260924 15:28:06.699934 13548 raft_consensus.cc:348] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.700088 13548 raft_consensus.cc:374] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.700152 13548 raft_consensus.cc:729] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.700273 13548 consensus_queue.cc:260] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.700539 13548 ts_tablet_manager.cc:1428] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:28:06.700574 13518 heartbeater.cc:503] Master 127.5.98.254:46827 was elected leader, sending a full tablet report...
W20260924 15:28:06.728139 13379 tablet.cc:2369] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:28:06.797312 13239 tablet.cc:2369] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:06.823637 13552 raft_consensus.cc:482] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:06.823804 13552 raft_consensus.cc:504] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.824216 13552 leader_election.cc:288] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:06.827296 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.824556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:46381 (local address 127.5.98.194:33021)
0924 15:28:06.824620 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.824623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.824761 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:28:06.824837 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.824838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.824849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.824918 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:06.825084 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.825544 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.826581 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.826944 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.827025 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.827034 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.827189 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.827228 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:28:06.827310 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.824472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:33021 (local address 127.5.98.193:46381)
0924 15:28:06.824668 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.824691 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.824775 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.824937 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.824938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.824940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:06.825010 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.825012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.825648 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.825649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.826486 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.826488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.827025 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.827026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.827109 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.827113 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.827181 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.827223 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":53,"thread_start_us":54,"threads_started":1}
I20260924 15:28:06.827399 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.824592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:37917 (local address 127.5.98.195:33135)
0924 15:28:06.824674 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.824676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.824687 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:28:06.824773 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.824775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.824787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.824855 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:06.825037 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.825553 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.826607 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.826996 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.827095 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.827104 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.827276 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.827318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:28:06.827561 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.824458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:33135 (local address 127.5.98.193:37917)
0924 15:28:06.824615 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:06.824628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:06.824695 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:06.824899 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:06.824900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:06.824903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:06.824966 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.824968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.825640 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.825641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.826511 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:06.826514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.827096 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.827097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:06.827178 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.827182 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:06.827266 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:28:06.827313 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260924 15:28:06.827862 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc466173c224a0ab3f52041b8040103" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" is_pre_election: true
I20260924 15:28:06.827919 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc466173c224a0ab3f52041b8040103" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba" is_pre_election: true
I20260924 15:28:06.828035 13332 raft_consensus.cc:2454] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:06.828034 13472 raft_consensus.cc:2454] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:06.828267 13126 leader_election.cc:302] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:06.828431 13552 raft_consensus.cc:2781] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:06.828480 13552 raft_consensus.cc:482] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:06.828509 13552 raft_consensus.cc:3037] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:06.829303 13552 raft_consensus.cc:504] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.829447 13552 leader_election.cc:288] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Requested vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:06.829578 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc466173c224a0ab3f52041b8040103" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
I20260924 15:28:06.829602 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc466173c224a0ab3f52041b8040103" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba"
I20260924 15:28:06.829664 13332 raft_consensus.cc:3037] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:06.829686 13472 raft_consensus.cc:3037] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:06.830375 13332 raft_consensus.cc:2454] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:06.830390 13472 raft_consensus.cc:2454] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:06.830540 13126 leader_election.cc:302] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:06.830619 13552 raft_consensus.cc:2781] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:06.830734 13552 raft_consensus.cc:686] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 LEADER]: Becoming Leader. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Running, Role: LEADER
I20260924 15:28:06.830842 13552 consensus_queue.cc:237] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } }
I20260924 15:28:06.831597 13035 catalog_manager.cc:5495] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } health_report { overall_health: HEALTHY } } }
I20260924 15:28:06.869596 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:06.866768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.5.98.193:44063)
0924 15:28:06.866837 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:06.866840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:06.866851 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:28:06.866913 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:06.866914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:06.866940 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:06.867032 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:06.867187 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.867806 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.868681 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:06.868911 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:06.868999 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:06.869194 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:06.869324 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:06.869423 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:06.869471 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 15:28:06.869508 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:28:06.871556 13332 raft_consensus.cc:1261] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:06.871966 13472 raft_consensus.cc:1261] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:06.871956 13557 consensus_queue.cc:1035] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:06.872272 13552 consensus_queue.cc:1035] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:06.874241 13560 mvcc.cc:205] Tried to move back new op lower bound from 7332920061422825472 to 7332920061259710464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920061422825472}]
I20260924 15:28:06.875003 13561 mvcc.cc:205] Tried to move back new op lower bound from 7332920061422825472 to 7332920061259710464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920061422825472 or (T in {7332920061422825472})}]
I20260924 15:28:06.875525 13562 mvcc.cc:205] Tried to move back new op lower bound from 7332920061422825472 to 7332920061259710464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920061422825472 or (T in {7332920061422825472})}]
I20260924 15:28:06.876426 13035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:06.876503 13035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:06.877276 13035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Sending DeleteTablet for 3 replicas of tablet 8bc466173c224a0ab3f52041b8040103
I20260924 15:28:06.877704 13172 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc466173c224a0ab3f52041b8040103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:06 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260924 15:28:06.877730 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc466173c224a0ab3f52041b8040103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:06 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 15:28:06.877947 13571 tablet_replica.cc:330] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: stopping tablet replica
I20260924 15:28:06.877938 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc466173c224a0ab3f52041b8040103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:06 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:49916
I20260924 15:28:06.878051 13571 raft_consensus.cc:2229] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:06.878162 13571 raft_consensus.cc:2258] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:06.878187 13572 tablet_replica.cc:330] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: stopping tablet replica
I20260924 15:28:06.878368 13572 raft_consensus.cc:2229] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:06.878402 13035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:06.878556 13035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:06.878630 13571 ts_tablet_manager.cc:1905] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:06.878757 13572 raft_consensus.cc:2258] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:06.879297 13572 ts_tablet_manager.cc:1905] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:06.880154 13172 tablet_service.cc:1467] Processing CreateTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 (DEFAULT_TABLE table=test-table [id=8158034346d94c52825a6fec503b0698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.880281 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15100e6fbc346d58b2da43eda08b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.880306 13571 ts_tablet_manager.cc:1918] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:06.880385 13571 log.cc:1196] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/8bc466173c224a0ab3f52041b8040103
I20260924 15:28:06.880695 13571 ts_tablet_manager.cc:1939] T 8bc466173c224a0ab3f52041b8040103 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting consensus metadata
I20260924 15:28:06.881701 13546 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Bootstrap starting.
I20260924 15:28:06.881805 13021 catalog_manager.cc:4841] TS f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021): tablet 8bc466173c224a0ab3f52041b8040103 (table test-table [id=d90c9b84ad0b4334a2d90f5577647f55]) successfully deleted
I20260924 15:28:06.882009 13452 tablet_service.cc:1467] Processing CreateTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 (DEFAULT_TABLE table=test-table [id=8158034346d94c52825a6fec503b0698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.882133 13546 tablet_bootstrap.cc:654] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.882155 13312 tablet_service.cc:1467] Processing CreateTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 (DEFAULT_TABLE table=test-table [id=8158034346d94c52825a6fec503b0698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:06.882248 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15100e6fbc346d58b2da43eda08b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.882653 13572 ts_tablet_manager.cc:1918] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:06.882769 13572 log.cc:1196] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/8bc466173c224a0ab3f52041b8040103
I20260924 15:28:06.882764 13546 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: No bootstrap required, opened a new log
I20260924 15:28:06.882812 13546 ts_tablet_manager.cc:1397] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:06.882977 13546 raft_consensus.cc:348] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.883062 13546 raft_consensus.cc:374] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.883090 13546 raft_consensus.cc:729] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.883140 13546 consensus_queue.cc:260] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.883231 13546 ts_tablet_manager.cc:1428] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:06.883309 13572 ts_tablet_manager.cc:1939] T 8bc466173c224a0ab3f52041b8040103 P 282450cf060e43ceaa8213f0a56fafba: Deleting consensus metadata
I20260924 15:28:06.883553 13547 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Bootstrap starting.
I20260924 15:28:06.883587 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15100e6fbc346d58b2da43eda08b4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:06.883994 13547 tablet_bootstrap.cc:654] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.884603 13547 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: No bootstrap required, opened a new log
I20260924 15:28:06.884655 13547 ts_tablet_manager.cc:1397] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:06.884704 13548 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Bootstrap starting.
I20260924 15:28:06.884790 13547 raft_consensus.cc:348] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.884886 13547 raft_consensus.cc:374] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.884908 13547 raft_consensus.cc:729] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d4dc0c49a0406ab103bbd8995ba1da, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.884953 13547 consensus_queue.cc:260] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.885037 13547 ts_tablet_manager.cc:1428] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:06.885121 13548 tablet_bootstrap.cc:654] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:06.885411 13570 tablet_replica.cc:330] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: stopping tablet replica
I20260924 15:28:06.885447 13022 catalog_manager.cc:4841] TS 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135): tablet 8bc466173c224a0ab3f52041b8040103 (table test-table [id=d90c9b84ad0b4334a2d90f5577647f55]) successfully deleted
I20260924 15:28:06.885521 13570 raft_consensus.cc:2229] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:06.885640 13570 raft_consensus.cc:2258] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:06.885963 13548 tablet_bootstrap.cc:492] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: No bootstrap required, opened a new log
I20260924 15:28:06.886011 13548 ts_tablet_manager.cc:1397] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:06.886092 13570 ts_tablet_manager.cc:1905] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:06.886160 13548 raft_consensus.cc:348] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.886227 13548 raft_consensus.cc:374] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:06.886257 13548 raft_consensus.cc:729] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Initialized, Role: FOLLOWER
I20260924 15:28:06.886307 13548 consensus_queue.cc:260] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:06.886392 13548 ts_tablet_manager.cc:1428] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 15:28:06.886595 13519 tablet.cc:2369] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:06.887580 13570 ts_tablet_manager.cc:1918] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:06.887642 13570 log.cc:1196] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/8bc466173c224a0ab3f52041b8040103
I20260924 15:28:06.887910 13570 ts_tablet_manager.cc:1939] T 8bc466173c224a0ab3f52041b8040103 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting consensus metadata
I20260924 15:28:06.888289 13022 catalog_manager.cc:4841] TS 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063): tablet 8bc466173c224a0ab3f52041b8040103 (table test-table [id=d90c9b84ad0b4334a2d90f5577647f55]) successfully deleted
I20260924 15:28:07.131378 13554 raft_consensus.cc:482] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:07.131485 13554 raft_consensus.cc:504] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.131855 13554 leader_election.cc:288] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063), f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021)
I20260924 15:28:07.134717 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.132047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:33021 (local address 127.5.98.195:40893)
0924 15:28:07.132123 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.132136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.132194 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.132369 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.132370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.132372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:07.132456 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.132458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.133140 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.133141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.133942 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.133945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.134465 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.134543 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.134550 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.134595 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.134625 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":28}
I20260924 15:28:07.134740 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.132095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:40893 (local address 127.5.98.194:33021)
0924 15:28:07.132141 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.132143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.132227 (+ 84us) server_negotiation.cc:407] Connection header received
0924 15:28:07.132270 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.132271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.132282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.132346 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:07.132532 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.133050 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134032 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.134380 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134443 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.134449 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.134615 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.134666 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:28:07.134786 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.132206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:60537 (local address 127.5.98.193:44063)
0924 15:28:07.132255 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.132257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.132287 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:28:07.132372 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.132373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.132382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.132446 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:07.132649 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.133120 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134054 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.134406 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134494 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.134502 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.134671 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.134713 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:28:07.135017 13576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.132118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44063 (local address 127.5.98.195:60537)
0924 15:28:07.132220 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.132233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.132301 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.132458 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.132459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.132461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:07.132575 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.132576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.133199 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.133200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.133953 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.133954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.134531 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.134533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.134595 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.134600 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.134657 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.134690 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260924 15:28:07.135007 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15100e6fbc346d58b2da43eda08b4f9" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" is_pre_election: true
I20260924 15:28:07.135128 13332 raft_consensus.cc:2454] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 0.
I20260924 15:28:07.135327 13405 leader_election.cc:302] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, f0d4dc0c49a0406ab103bbd8995ba1da; no voters:
I20260924 15:28:07.135289 13192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15100e6fbc346d58b2da43eda08b4f9" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a314c131a46abb4e0f5aaeea6dab7" is_pre_election: true
I20260924 15:28:07.135429 13192 raft_consensus.cc:2454] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 0.
I20260924 15:28:07.135460 13554 raft_consensus.cc:2781] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:07.135496 13554 raft_consensus.cc:482] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:07.135528 13554 raft_consensus.cc:3037] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.136276 13554 raft_consensus.cc:504] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.136433 13554 leader_election.cc:288] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 election: Requested vote from peers 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063), f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021)
I20260924 15:28:07.136574 13192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15100e6fbc346d58b2da43eda08b4f9" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a314c131a46abb4e0f5aaeea6dab7"
I20260924 15:28:07.136658 13192 raft_consensus.cc:3037] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.136618 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b15100e6fbc346d58b2da43eda08b4f9" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
I20260924 15:28:07.136695 13332 raft_consensus.cc:3037] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.137346 13332 raft_consensus.cc:2454] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 1.
I20260924 15:28:07.137362 13192 raft_consensus.cc:2454] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 1.
I20260924 15:28:07.137548 13406 leader_election.cc:302] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:07.137652 13554 raft_consensus.cc:2781] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:07.137741 13554 raft_consensus.cc:686] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 1 LEADER]: Becoming Leader. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Running, Role: LEADER
I20260924 15:28:07.137809 13554 consensus_queue.cc:237] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.138509 13035 catalog_manager.cc:5495] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba reported cstate change: term changed from 0 to 1, leader changed from <none> to 282450cf060e43ceaa8213f0a56fafba (127.5.98.195). New cstate: current_term: 1 leader_uuid: "282450cf060e43ceaa8213f0a56fafba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } health_report { overall_health: HEALTHY } } }
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263686, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.254@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:07.199007 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.192399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.5.98.254:46827)
0924 15:28:07.192491 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.192492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.192548 (+ 56us) server_negotiation.cc:407] Connection header received
0924 15:28:07.192610 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.192611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.192621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.192681 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:07.193096 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.193583 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.194255 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.194437 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.194486 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.196604 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:07.196606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:07.196606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:07.196612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:07.198024 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:07.198364 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:07.198365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:07.198365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:07.198415 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:07.198573 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:07.198573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:07.198574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:07.198667 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:07.198714 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.198830 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.198935 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
I20260924 15:28:07.199141 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.192336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:46827 (local address 127.0.0.1:53494)
0924 15:28:07.192456 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.192466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.192555 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.192695 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.192696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.192983 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:07.193031 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.193033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.193666 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.193666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.194191 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.194192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.194496 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.194497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.194543 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.195092 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:07.195097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:07.196493 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:07.198098 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:07.198105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:07.198107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:07.198300 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:07.198471 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:07.198472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:07.198472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:07.198527 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:07.198716 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:07.198722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:07.198763 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.198814 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.198999 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":61,"thread_start_us":65,"threads_started":1}
I20260924 15:28:07.201182 13035 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:53494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:28:07.201376 13035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:07.202696 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c57e1e8c9d474fa5a6f752c35bb217d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:28:07.202704 13172 tablet_service.cc:1467] Processing CreateTablet for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c57e1e8c9d474fa5a6f752c35bb217d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:28:07.202693 13452 tablet_service.cc:1467] Processing CreateTablet for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c57e1e8c9d474fa5a6f752c35bb217d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:28:07.202842 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.202842 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.202853 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c4c1296a2b4b7f898f3afeb4a2b9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.204106 13547 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: Bootstrap starting.
I20260924 15:28:07.204363 13546 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: Bootstrap starting.
I20260924 15:28:07.204519 13548 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: Bootstrap starting.
I20260924 15:28:07.204560 13547 tablet_bootstrap.cc:654] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.204691 13546 tablet_bootstrap.cc:654] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.204942 13548 tablet_bootstrap.cc:654] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.205160 13547 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: No bootstrap required, opened a new log
I20260924 15:28:07.205219 13547 ts_tablet_manager.cc:1397] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:07.205371 13547 raft_consensus.cc:348] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205462 13547 raft_consensus.cc:374] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.205488 13547 raft_consensus.cc:729] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d4dc0c49a0406ab103bbd8995ba1da, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.205538 13547 consensus_queue.cc:260] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205579 13548 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: No bootstrap required, opened a new log
I20260924 15:28:07.205570 13546 tablet_bootstrap.cc:492] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: No bootstrap required, opened a new log
I20260924 15:28:07.205619 13548 ts_tablet_manager.cc:1397] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:07.205624 13546 ts_tablet_manager.cc:1397] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:07.205637 13547 ts_tablet_manager.cc:1428] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.205741 13546 raft_consensus.cc:348] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205756 13548 raft_consensus.cc:348] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205802 13546 raft_consensus.cc:374] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.205816 13546 raft_consensus.cc:729] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.205818 13548 raft_consensus.cc:374] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.205844 13548 raft_consensus.cc:729] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.205864 13546 consensus_queue.cc:260] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205893 13548 consensus_queue.cc:260] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.205956 13546 ts_tablet_manager.cc:1428] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.205986 13548 ts_tablet_manager.cc:1428] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.205986 13557 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.206081 13557 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.206034 13554 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.206127 13554 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.206094 13553 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.206171 13553 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.307363 13557 raft_consensus.cc:482] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:07.307477 13557 raft_consensus.cc:504] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.307678 13557 leader_election.cc:288] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:07.307940 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c4c1296a2b4b7f898f3afeb4a2b9fb" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" is_pre_election: true
I20260924 15:28:07.307940 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c4c1296a2b4b7f898f3afeb4a2b9fb" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba" is_pre_election: true
I20260924 15:28:07.308049 13332 raft_consensus.cc:2454] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:07.308049 13472 raft_consensus.cc:2454] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:07.308195 13125 leader_election.cc:302] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774a314c131a46abb4e0f5aaeea6dab7, f0d4dc0c49a0406ab103bbd8995ba1da; no voters:
I20260924 15:28:07.308318 13557 raft_consensus.cc:2781] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:07.308370 13557 raft_consensus.cc:482] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:07.308385 13557 raft_consensus.cc:3037] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.309077 13557 raft_consensus.cc:504] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.309206 13557 leader_election.cc:288] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Requested vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:07.309509 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c4c1296a2b4b7f898f3afeb4a2b9fb" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba"
I20260924 15:28:07.309510 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c4c1296a2b4b7f898f3afeb4a2b9fb" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
I20260924 15:28:07.309592 13472 raft_consensus.cc:3037] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.309599 13332 raft_consensus.cc:3037] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.310335 13472 raft_consensus.cc:2454] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:07.310335 13332 raft_consensus.cc:2454] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:07.310501 13125 leader_election.cc:302] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774a314c131a46abb4e0f5aaeea6dab7, f0d4dc0c49a0406ab103bbd8995ba1da; no voters:
I20260924 15:28:07.310614 13557 raft_consensus.cc:2781] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:07.310669 13557 raft_consensus.cc:686] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 LEADER]: Becoming Leader. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Running, Role: LEADER
I20260924 15:28:07.310716 13557 consensus_queue.cc:237] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.310985 13566 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: TxnStatusTablet state changed. Reason: New leader 774a314c131a46abb4e0f5aaeea6dab7. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.311084 13566 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:28:07.311242 13590 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:28:07.311347 13590 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:28:07.311352 13035 catalog_manager.cc:5495] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } health_report { overall_health: UNKNOWN } } }
I20260924 15:28:07.322783 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.319902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44063 (local address 127.0.0.1:40936)
0924 15:28:07.319982 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.319994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.320045 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.320303 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.320304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.320305 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:28:07.320349 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.320351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.321221 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.321222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.321830 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.321832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.322336 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.322337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.322436 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.322460 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.322646 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.322653 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.322689 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.322731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:28:07.322781 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.320122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.5.98.193:44063)
0924 15:28:07.320181 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.320183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.320193 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:07.320215 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.320216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.320227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.320288 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.320405 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.321138 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.321949 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.322250 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.322321 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.322544 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.322579 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.322639 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.322692 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.322713 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:28:07.323791 13472 raft_consensus.cc:1261] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.323812 13332 raft_consensus.cc:1261] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.323987 13566 consensus_queue.cc:1035] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.324087 13557 consensus_queue.cc:1035] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.324615 13553 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: TxnStatusTablet state changed. Reason: New leader 774a314c131a46abb4e0f5aaeea6dab7. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.324633 13554 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: TxnStatusTablet state changed. Reason: New leader 774a314c131a46abb4e0f5aaeea6dab7. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.324713 13554 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.324714 13553 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.325191 13552 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.325264 13552 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:28:07.325369 13553 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.325453 13553 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P f0d4dc0c49a0406ab103bbd8995ba1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.326195 13552 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.326272 13552 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 774a314c131a46abb4e0f5aaeea6dab7: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:28:07.326982 13554 tablet_replica.cc:439] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } } }
I20260924 15:28:07.327062 13554 tablet_replica.cc:442] T 83c4c1296a2b4b7f898f3afeb4a2b9fb P 282450cf060e43ceaa8213f0a56fafba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:28:07.329985 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.327220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.5.98.195:33135)
0924 15:28:07.327257 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.327259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.327269 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:07.327355 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.327356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.327376 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.327454 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.327621 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.328354 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.329085 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.329402 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.329464 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.329613 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.329736 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.329811 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.329883 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.329904 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:28:07.335384 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.332419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:45943 (local address 127.5.98.193:44063)
0924 15:28:07.332460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.332462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.332571 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:28:07.332626 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.332626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.332636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.332687 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.332781 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.333445 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.334666 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.334957 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.335019 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.335174 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.335205 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.335259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.335302 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.335321 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:28:07.335417 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.332373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44063 (local address 127.5.98.195:45943)
0924 15:28:07.332503 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.332513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.332566 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.332694 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.332695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.332696 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:28:07.332740 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.332741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.333511 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.333512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.334588 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.334591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.335035 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.335036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.335085 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.335097 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.335264 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.335265 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.335299 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.335330 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260924 15:28:07.337683 13332 raft_consensus.cc:1261] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Refusing update from remote peer 282450cf060e43ceaa8213f0a56fafba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.337761 13192 raft_consensus.cc:1261] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Refusing update from remote peer 282450cf060e43ceaa8213f0a56fafba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.337919 13554 consensus_queue.cc:1035] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.338032 13577 consensus_queue.cc:1035] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Connected to new peer: Peer: permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.353060 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.342972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:59477 (local address 127.5.98.195:33135)
0924 15:28:07.343012 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.343014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.350532 (+ 7518us) server_negotiation.cc:407] Connection header received
0924 15:28:07.350559 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.350560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.350571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.350647 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.350770 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.351220 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.352306 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.352606 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.352645 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.352850 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.352884 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.352921 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.352997 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.353013 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:28:07.353106 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.342814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:33135 (local address 127.5.98.193:59477)
0924 15:28:07.350432 (+ 7618us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.350444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:07.350506 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:07.350659 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:07.350660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:07.350661 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:28:07.350703 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.350705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.351285 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.351286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.352205 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:07.352208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.352699 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.352700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:07.352754 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.352769 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.352939 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.352941 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:28:07.352990 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:28:07.353032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7574,"thread_start_us":78,"threads_started":1}
I20260924 15:28:07.362329 13031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.362437 13031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.363067 13031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Sending DeleteTablet for 3 replicas of tablet b15100e6fbc346d58b2da43eda08b4f9
I20260924 15:28:07.363302 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 15:28:07.363390 13571 tablet_replica.cc:330] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: stopping tablet replica
I20260924 15:28:07.363427 13571 raft_consensus.cc:2229] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.363458 13571 raft_consensus.cc:2258] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.363432 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:49916
I20260924 15:28:07.363512 13572 tablet_replica.cc:330] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: stopping tablet replica
I20260924 15:28:07.363554 13572 raft_consensus.cc:2229] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:07.363627 13572 raft_consensus.cc:2258] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.363718 13172 tablet_service.cc:1514] Processing DeleteTablet for tablet b15100e6fbc346d58b2da43eda08b4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260924 15:28:07.363806 13570 tablet_replica.cc:330] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: stopping tablet replica
I20260924 15:28:07.363796 13571 ts_tablet_manager.cc:1905] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.363863 13570 raft_consensus.cc:2229] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.363902 13570 raft_consensus.cc:2258] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.363963 13572 ts_tablet_manager.cc:1905] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.365439 13571 ts_tablet_manager.cc:1918] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:28:07.365494 13571 log.cc:1196] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/b15100e6fbc346d58b2da43eda08b4f9
I20260924 15:28:07.365746 13571 ts_tablet_manager.cc:1939] T b15100e6fbc346d58b2da43eda08b4f9 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting consensus metadata
I20260924 15:28:07.365777 13572 ts_tablet_manager.cc:1918] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:28:07.365834 13572 log.cc:1196] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/b15100e6fbc346d58b2da43eda08b4f9
I20260924 15:28:07.366062 13572 ts_tablet_manager.cc:1939] T b15100e6fbc346d58b2da43eda08b4f9 P 282450cf060e43ceaa8213f0a56fafba: Deleting consensus metadata
I20260924 15:28:07.366060 13021 catalog_manager.cc:4841] TS f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021): tablet b15100e6fbc346d58b2da43eda08b4f9 (table test-table [id=8158034346d94c52825a6fec503b0698]) successfully deleted
I20260924 15:28:07.366446 13022 catalog_manager.cc:4841] TS 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135): tablet b15100e6fbc346d58b2da43eda08b4f9 (table test-table [id=8158034346d94c52825a6fec503b0698]) successfully deleted
I20260924 15:28:07.366499 13570 ts_tablet_manager.cc:1905] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.368012 13570 ts_tablet_manager.cc:1918] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:28:07.368063 13570 log.cc:1196] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/b15100e6fbc346d58b2da43eda08b4f9
I20260924 15:28:07.368261 13570 ts_tablet_manager.cc:1939] T b15100e6fbc346d58b2da43eda08b4f9 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting consensus metadata
I20260924 15:28:07.368705 13022 catalog_manager.cc:4841] TS 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063): tablet b15100e6fbc346d58b2da43eda08b4f9 (table test-table [id=8158034346d94c52825a6fec503b0698]) successfully deleted
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[12990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:28:07.415215 13585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:07.398619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.5.98.254:46827)
0924 15:28:07.398667 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.398670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.398820 (+ 150us) server_negotiation.cc:407] Connection header received
0924 15:28:07.398887 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.398889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.398902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.398961 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:07.399725 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.400570 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.401682 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.401994 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.402069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.415146 (+ 13077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53508: BlockingRecv error: recv got EOF from 127.0.0.1:53508 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[12990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM
I20260924 15:28:07.425257 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.415997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.5.98.254:46827)
0924 15:28:07.416038 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.416040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.416273 (+ 233us) server_negotiation.cc:407] Connection header received
0924 15:28:07.416326 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.416327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.416338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.416400 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:07.416960 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.417714 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.418700 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.419026 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.419101 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.422096 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:07.422098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:07.422098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:07.422135 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:07.423906 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:07.424319 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:07.424320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:07.424321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:07.424387 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:07.424610 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:07.424611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:07.424612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:07.424749 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:07.424856 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.425037 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.425119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:07.427011 13031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:07.427191 13031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:07.428596 13172 tablet_service.cc:1467] Processing CreateTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 (DEFAULT_TABLE table=test-table [id=9285c6c190844782a6570640903869ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.428723 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19426f0fe384e19913ed11c3ecbcc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.430090 13546 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Bootstrap starting.
I20260924 15:28:07.430100 13452 tablet_service.cc:1467] Processing CreateTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 (DEFAULT_TABLE table=test-table [id=9285c6c190844782a6570640903869ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.430218 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19426f0fe384e19913ed11c3ecbcc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.430541 13546 tablet_bootstrap.cc:654] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.431509 13548 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Bootstrap starting.
I20260924 15:28:07.431627 13546 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: No bootstrap required, opened a new log
I20260924 15:28:07.431725 13546 ts_tablet_manager.cc:1397] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:07.431879 13546 raft_consensus.cc:348] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.431943 13548 tablet_bootstrap.cc:654] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.431945 13546 raft_consensus.cc:374] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.431978 13546 raft_consensus.cc:729] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.432026 13546 consensus_queue.cc:260] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.432124 13546 ts_tablet_manager.cc:1428] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.432355 13312 tablet_service.cc:1467] Processing CreateTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 (DEFAULT_TABLE table=test-table [id=9285c6c190844782a6570640903869ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.432471 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19426f0fe384e19913ed11c3ecbcc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.433761 13547 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Bootstrap starting.
I20260924 15:28:07.434208 13547 tablet_bootstrap.cc:654] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.434808 13548 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: No bootstrap required, opened a new log
I20260924 15:28:07.434867 13548 ts_tablet_manager.cc:1397] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:28:07.434990 13548 raft_consensus.cc:348] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.435066 13548 raft_consensus.cc:374] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.435086 13548 raft_consensus.cc:729] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.435156 13548 consensus_queue.cc:260] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.435247 13548 ts_tablet_manager.cc:1428] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.435618 13547 tablet_bootstrap.cc:492] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: No bootstrap required, opened a new log
I20260924 15:28:07.435705 13547 ts_tablet_manager.cc:1397] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:28:07.435854 13547 raft_consensus.cc:348] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.435914 13547 raft_consensus.cc:374] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.435933 13547 raft_consensus.cc:729] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d4dc0c49a0406ab103bbd8995ba1da, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.435994 13547 consensus_queue.cc:260] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.436084 13547 ts_tablet_manager.cc:1428] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.472678 13566 raft_consensus.cc:482] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:07.472819 13566 raft_consensus.cc:504] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.473017 13566 leader_election.cc:288] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:07.473290 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19426f0fe384e19913ed11c3ecbcc76" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" is_pre_election: true
I20260924 15:28:07.473289 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19426f0fe384e19913ed11c3ecbcc76" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba" is_pre_election: true
I20260924 15:28:07.473392 13472 raft_consensus.cc:2454] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:07.473407 13332 raft_consensus.cc:2454] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 0.
I20260924 15:28:07.473548 13126 leader_election.cc:302] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:07.473654 13566 raft_consensus.cc:2781] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:07.473706 13566 raft_consensus.cc:482] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:07.473740 13566 raft_consensus.cc:3037] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.474454 13566 raft_consensus.cc:504] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.474614 13566 leader_election.cc:288] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Requested vote from peers f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021), 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135)
I20260924 15:28:07.474777 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19426f0fe384e19913ed11c3ecbcc76" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
I20260924 15:28:07.474879 13332 raft_consensus.cc:3037] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.474867 13472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19426f0fe384e19913ed11c3ecbcc76" candidate_uuid: "774a314c131a46abb4e0f5aaeea6dab7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282450cf060e43ceaa8213f0a56fafba"
I20260924 15:28:07.474968 13472 raft_consensus.cc:3037] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.475575 13332 raft_consensus.cc:2454] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:07.475769 13125 leader_election.cc:302] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774a314c131a46abb4e0f5aaeea6dab7, f0d4dc0c49a0406ab103bbd8995ba1da; no voters:
I20260924 15:28:07.475855 13566 raft_consensus.cc:2781] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:07.475898 13566 raft_consensus.cc:686] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 LEADER]: Becoming Leader. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Running, Role: LEADER
I20260924 15:28:07.475896 13472 raft_consensus.cc:2454] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774a314c131a46abb4e0f5aaeea6dab7 in term 1.
I20260924 15:28:07.475946 13566 consensus_queue.cc:237] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.476562 13031 catalog_manager.cc:5495] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "774a314c131a46abb4e0f5aaeea6dab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } health_report { overall_health: UNKNOWN } } }
I20260924 15:28:07.483326 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.480461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.5.98.193:44063)
0924 15:28:07.480504 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.480506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.480557 (+ 51us) server_negotiation.cc:407] Connection header received
0924 15:28:07.480581 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.480581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.480598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.480644 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.480746 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.481527 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.482667 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.482849 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.482899 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.483078 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.483108 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.483146 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.483213 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.483232 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:28:07.484081 13332 raft_consensus.cc:1261] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.484083 13472 raft_consensus.cc:1261] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Refusing update from remote peer 774a314c131a46abb4e0f5aaeea6dab7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.484306 13566 consensus_queue.cc:1035] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.484400 13557 consensus_queue.cc:1035] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.487026 13035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.487128 13035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.487758 13035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Sending DeleteTablet for 3 replicas of tablet b19426f0fe384e19913ed11c3ecbcc76
I20260924 15:28:07.488061 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 15:28:07.488060 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:49916
I20260924 15:28:07.488150 13572 tablet_replica.cc:330] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: stopping tablet replica
I20260924 15:28:07.488149 13571 tablet_replica.cc:330] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: stopping tablet replica
I20260924 15:28:07.488195 13571 raft_consensus.cc:2229] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.488200 13572 raft_consensus.cc:2229] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.488237 13571 raft_consensus.cc:2258] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.488241 13572 raft_consensus.cc:2258] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.488251 13035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:07.488402 13035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:07.488513 13172 tablet_service.cc:1514] Processing DeleteTablet for tablet b19426f0fe384e19913ed11c3ecbcc76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260924 15:28:07.488615 13570 tablet_replica.cc:330] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: stopping tablet replica
I20260924 15:28:07.488679 13570 raft_consensus.cc:2229] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:07.488737 13570 raft_consensus.cc:2258] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.488830 13572 ts_tablet_manager.cc:1905] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.489300 13570 ts_tablet_manager.cc:1905] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.490219 13571 ts_tablet_manager.cc:1905] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.490257 13572 ts_tablet_manager.cc:1918] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:07.490301 13572 log.cc:1196] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/b19426f0fe384e19913ed11c3ecbcc76
I20260924 15:28:07.490521 13572 ts_tablet_manager.cc:1939] T b19426f0fe384e19913ed11c3ecbcc76 P 282450cf060e43ceaa8213f0a56fafba: Deleting consensus metadata
I20260924 15:28:07.490777 13570 ts_tablet_manager.cc:1918] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:07.490828 13570 log.cc:1196] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/b19426f0fe384e19913ed11c3ecbcc76
I20260924 15:28:07.490832 13022 catalog_manager.cc:4841] TS 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135): tablet b19426f0fe384e19913ed11c3ecbcc76 (table test-table [id=9285c6c190844782a6570640903869ca]) successfully deleted
I20260924 15:28:07.491055 13570 ts_tablet_manager.cc:1939] T b19426f0fe384e19913ed11c3ecbcc76 P 774a314c131a46abb4e0f5aaeea6dab7: Deleting consensus metadata
I20260924 15:28:07.491452 13172 tablet_service.cc:1467] Processing CreateTablet for tablet 675952d42b6b4029a122267e7fcc1564 (DEFAULT_TABLE table=test-table [id=c699bd2880e54abc85c8d5ed21a85246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.491467 13452 tablet_service.cc:1467] Processing CreateTablet for tablet 675952d42b6b4029a122267e7fcc1564 (DEFAULT_TABLE table=test-table [id=c699bd2880e54abc85c8d5ed21a85246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.491572 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 675952d42b6b4029a122267e7fcc1564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.491564 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 675952d42b6b4029a122267e7fcc1564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.491730 13022 catalog_manager.cc:4841] TS 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063): tablet b19426f0fe384e19913ed11c3ecbcc76 (table test-table [id=9285c6c190844782a6570640903869ca]) successfully deleted
I20260924 15:28:07.491789 13571 ts_tablet_manager.cc:1918] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:07.491839 13571 log.cc:1196] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/b19426f0fe384e19913ed11c3ecbcc76
I20260924 15:28:07.492038 13571 ts_tablet_manager.cc:1939] T b19426f0fe384e19913ed11c3ecbcc76 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting consensus metadata
I20260924 15:28:07.492690 13548 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Bootstrap starting.
I20260924 15:28:07.492967 13021 catalog_manager.cc:4841] TS f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021): tablet b19426f0fe384e19913ed11c3ecbcc76 (table test-table [id=9285c6c190844782a6570640903869ca]) successfully deleted
I20260924 15:28:07.493064 13546 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: Bootstrap starting.
I20260924 15:28:07.493068 13548 tablet_bootstrap.cc:654] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.493355 13546 tablet_bootstrap.cc:654] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.493582 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 675952d42b6b4029a122267e7fcc1564 (DEFAULT_TABLE table=test-table [id=c699bd2880e54abc85c8d5ed21a85246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:07.493698 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 675952d42b6b4029a122267e7fcc1564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:07.493887 13546 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: No bootstrap required, opened a new log
I20260924 15:28:07.493944 13546 ts_tablet_manager.cc:1397] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:07.494061 13546 raft_consensus.cc:348] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.494132 13546 raft_consensus.cc:374] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.494155 13546 raft_consensus.cc:729] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774a314c131a46abb4e0f5aaeea6dab7, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.494194 13546 consensus_queue.cc:260] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.494292 13546 ts_tablet_manager.cc:1428] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.494871 13547 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Bootstrap starting.
I20260924 15:28:07.495257 13547 tablet_bootstrap.cc:654] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:07.495280 13548 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: No bootstrap required, opened a new log
I20260924 15:28:07.495338 13548 ts_tablet_manager.cc:1397] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:28:07.495487 13548 raft_consensus.cc:348] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.495563 13548 raft_consensus.cc:374] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.495597 13548 raft_consensus.cc:729] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.495646 13548 consensus_queue.cc:260] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.495795 13548 ts_tablet_manager.cc:1428] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:07.496109 13547 tablet_bootstrap.cc:492] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: No bootstrap required, opened a new log
I20260924 15:28:07.496177 13547 ts_tablet_manager.cc:1397] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:07.496315 13547 raft_consensus.cc:348] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.496381 13547 raft_consensus.cc:374] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:07.496404 13547 raft_consensus.cc:729] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d4dc0c49a0406ab103bbd8995ba1da, State: Initialized, Role: FOLLOWER
I20260924 15:28:07.496451 13547 consensus_queue.cc:260] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.496536 13547 ts_tablet_manager.cc:1428] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:07.756706 13554 raft_consensus.cc:482] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:07.756805 13554 raft_consensus.cc:504] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.756963 13554 leader_election.cc:288] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063), f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021)
I20260924 15:28:07.757223 13192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "675952d42b6b4029a122267e7fcc1564" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a314c131a46abb4e0f5aaeea6dab7" is_pre_election: true
I20260924 15:28:07.757221 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "675952d42b6b4029a122267e7fcc1564" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" is_pre_election: true
I20260924 15:28:07.757328 13192 raft_consensus.cc:2454] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 0.
I20260924 15:28:07.757325 13332 raft_consensus.cc:2454] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 0.
I20260924 15:28:07.757545 13406 leader_election.cc:302] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:07.757686 13554 raft_consensus.cc:2781] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:07.757738 13554 raft_consensus.cc:482] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:07.757753 13554 raft_consensus.cc:3037] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.758428 13554 raft_consensus.cc:504] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.758561 13554 leader_election.cc:288] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 election: Requested vote from peers 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063), f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021)
I20260924 15:28:07.758700 13332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "675952d42b6b4029a122267e7fcc1564" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da"
I20260924 15:28:07.758711 13192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "675952d42b6b4029a122267e7fcc1564" candidate_uuid: "282450cf060e43ceaa8213f0a56fafba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774a314c131a46abb4e0f5aaeea6dab7"
I20260924 15:28:07.758778 13192 raft_consensus.cc:3037] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.758791 13332 raft_consensus.cc:3037] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:07.759362 13192 raft_consensus.cc:2454] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 1.
I20260924 15:28:07.759485 13332 raft_consensus.cc:2454] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282450cf060e43ceaa8213f0a56fafba in term 1.
I20260924 15:28:07.759541 13406 leader_election.cc:302] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282450cf060e43ceaa8213f0a56fafba, 774a314c131a46abb4e0f5aaeea6dab7; no voters:
I20260924 15:28:07.759617 13554 raft_consensus.cc:2781] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:07.759689 13554 raft_consensus.cc:686] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 1 LEADER]: Becoming Leader. State: Replica: 282450cf060e43ceaa8213f0a56fafba, State: Running, Role: LEADER
I20260924 15:28:07.759748 13554 consensus_queue.cc:237] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } }
I20260924 15:28:07.760285 13035 catalog_manager.cc:5495] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba reported cstate change: term changed from 0 to 1, leader changed from <none> to 282450cf060e43ceaa8213f0a56fafba (127.5.98.195). New cstate: current_term: 1 leader_uuid: "282450cf060e43ceaa8213f0a56fafba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282450cf060e43ceaa8213f0a56fafba" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 33135 } health_report { overall_health: HEALTHY } } }
I20260924 15:28:07.781932 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.779796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.5.98.195:33135)
0924 15:28:07.779841 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:07.779843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:07.779854 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:28:07.779920 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:07.779921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:07.779931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:07.779990 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:07.780081 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.780609 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.781273 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:07.781494 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:07.781553 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:07.781722 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:07.781759 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:07.781817 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:07.781853 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 15:28:07.781870 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:28:07.784119 13332 raft_consensus.cc:1261] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Refusing update from remote peer 282450cf060e43ceaa8213f0a56fafba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.784149 13192 raft_consensus.cc:1261] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Refusing update from remote peer 282450cf060e43ceaa8213f0a56fafba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:07.784322 13554 consensus_queue.cc:1035] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d4dc0c49a0406ab103bbd8995ba1da" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.784462 13577 consensus_queue.cc:1035] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [LEADER]: Connected to new peer: Peer: permanent_uuid: "774a314c131a46abb4e0f5aaeea6dab7" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:07.794586 13031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.794691 13031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:07.795300 13031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fa2b99fda5428ca1d981f9c4597cf6: Sending DeleteTablet for 3 replicas of tablet 675952d42b6b4029a122267e7fcc1564
I20260924 15:28:07.795603 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet 675952d42b6b4029a122267e7fcc1564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42224
I20260924 15:28:07.795632 13452 tablet_service.cc:1514] Processing DeleteTablet for tablet 675952d42b6b4029a122267e7fcc1564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:49916
I20260924 15:28:07.795643 13172 tablet_service.cc:1514] Processing DeleteTablet for tablet 675952d42b6b4029a122267e7fcc1564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:07 UTC) from {username='oryx', principal='oryx/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:40916
I20260924 15:28:07.795730 13571 tablet_replica.cc:330] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: stopping tablet replica
I20260924 15:28:07.795781 13571 raft_consensus.cc:2229] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.795821 13571 raft_consensus.cc:2258] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.795900 13572 tablet_replica.cc:330] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: stopping tablet replica
I20260924 15:28:07.795958 13572 raft_consensus.cc:2229] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:07.796020 13572 raft_consensus.cc:2258] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.796103 13570 tablet_replica.cc:330] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7: stopping tablet replica
I20260924 15:28:07.796149 13571 ts_tablet_manager.cc:1905] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.796169 13570 raft_consensus.cc:2229] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:07.796201 13570 raft_consensus.cc:2258] T 675952d42b6b4029a122267e7fcc1564 P 774a314c131a46abb4e0f5aaeea6dab7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:07.796454 13572 ts_tablet_manager.cc:1905] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:07.796595 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13107
I20260924 15:28:07.797958 13572 ts_tablet_manager.cc:1918] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:28:07.797947 13571 ts_tablet_manager.cc:1918] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:28:07.798038 13571 log.cc:1196] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/675952d42b6b4029a122267e7fcc1564
I20260924 15:28:07.798040 13572 log.cc:1196] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/675952d42b6b4029a122267e7fcc1564
I20260924 15:28:07.798295 13572 ts_tablet_manager.cc:1939] T 675952d42b6b4029a122267e7fcc1564 P 282450cf060e43ceaa8213f0a56fafba: Deleting consensus metadata
I20260924 15:28:07.798324 13571 ts_tablet_manager.cc:1939] T 675952d42b6b4029a122267e7fcc1564 P f0d4dc0c49a0406ab103bbd8995ba1da: Deleting consensus metadata
I20260924 15:28:07.799134 13021 catalog_manager.cc:4841] TS f0d4dc0c49a0406ab103bbd8995ba1da (127.5.98.194:33021): tablet 675952d42b6b4029a122267e7fcc1564 (table test-table [id=c699bd2880e54abc85c8d5ed21a85246]) successfully deleted
I20260924 15:28:07.799763 13022 catalog_manager.cc:4841] TS 282450cf060e43ceaa8213f0a56fafba (127.5.98.195:33135): tablet 675952d42b6b4029a122267e7fcc1564 (table test-table [id=c699bd2880e54abc85c8d5ed21a85246]) successfully deleted
W20260924 15:28:07.804610 13022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:44063 (error 108)
I20260924 15:28:07.804916 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13247
I20260924 15:28:07.805992 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:07.805050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44063 (local address 127.0.0.1:40946)
0924 15:28:07.805900 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:07.805955 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:44063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":63,"threads_started":1}
W20260924 15:28:07.806110 13022 catalog_manager.cc:4584] TS 774a314c131a46abb4e0f5aaeea6dab7 (127.5.98.193:44063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 675952d42b6b4029a122267e7fcc1564: Network error: Client connection negotiation failed: client connection to 127.5.98.193:44063: connect: Connection refused (error 111)
I20260924 15:28:07.810364 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13387
I20260924 15:28:07.816648 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13003
2026-09-24T15:28:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2999 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[13635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[13635](info): set up 2 sockets
Sep 24 15:28:07 dist-test-slave-4g6f krb5kdc[13635](info): commencing operation
krb5kdc: starting...
W20260924 15:28:08.861316 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:08 dist-test-slave-4g6f krb5kdc[13635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:28:08.978638 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33643
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:33643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:09.060695 13648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:09.060927 13648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:09.060964 13648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:09.062146 13648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:09.062208 13648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:09.062235 13648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:09.062263 13648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:09.062291 13648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:09.063874 13648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:33643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:09.064147 13648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:09.064368 13648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:09.069101 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:09.069216 13656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:09.069149 13648 server_base.cc:1034] running on GCE node
W20260924 15:28:09.069108 13653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:09.069674 13648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:09.069973 13648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:09.071111 13648 hybrid_clock.cc:648] HybridClock initialized: now 1790263689071102 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T15:28:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:09 dist-test-slave-4g6f krb5kdc[13661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:09 dist-test-slave-4g6f krb5kdc[13661](info): set up 2 sockets
Sep 24 15:28:09 dist-test-slave-4g6f krb5kdc[13661](info): commencing operation
krb5kdc: starting...
W20260924 15:28:10.113456 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:28:10.229736 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41601
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:10.310353 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:10.310595 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:10.310629 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:10.311815 13674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:10.311870 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:10.311893 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:10.311914 13674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:10.311945 13674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:10.313380 13674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:10.313656 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:10.313874 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:10.318646 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.318661 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.318668 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:10.318920 13674 server_base.cc:1034] running on GCE node
I20260924 15:28:10.319185 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:10.319492 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:10.320683 13674 hybrid_clock.cc:648] HybridClock initialized: now 1790263690320653 us; error 42 us; skew 500 ppm
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:10.323166 13674 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:28:10.323619 13674 webserver.cc:460] Webserver started at http://127.5.98.254:35023/ using document root <none> and password file <none>
I20260924 15:28:10.323884 13674 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:10.323935 13674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:10.324036 13674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:10.325062 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "d7513f8505fb409fbe5083b9e16b851e"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "0e9af8e77655ee3537cdab020bac1ee5"
server_key_iv: "f818b4cda4f539750b4b85e10f343e33"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.325388 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "d7513f8505fb409fbe5083b9e16b851e"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "0e9af8e77655ee3537cdab020bac1ee5"
server_key_iv: "f818b4cda4f539750b4b85e10f343e33"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.326682 13674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:10.327358 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.327529 13674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:10.327617 13674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "d7513f8505fb409fbe5083b9e16b851e"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "0e9af8e77655ee3537cdab020bac1ee5"
server_key_iv: "f818b4cda4f539750b4b85e10f343e33"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.327718 13674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:10.335729 13674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:10.337663 13674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:10.337796 13674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:10.341747 13674 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:41601
I20260924 15:28:10.341820 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:41601 every 8 connection(s)
I20260924 15:28:10.342139 13674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:10.343015 13742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:10.344301 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13674
I20260924 15:28:10.344409 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:10.344624 5515 external_mini_cluster.cc:1428] Setting key 24b0d2cd5c7fc41f1de78128218634cf
I20260924 15:28:10.345942 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Bootstrap starting.
I20260924 15:28:10.346690 13742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:10.347000 13742 log.cc:824] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:10.347744 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: No bootstrap required, opened a new log
I20260924 15:28:10.350116 13742 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:10.350317 13742 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:10.350356 13742 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7513f8505fb409fbe5083b9e16b851e, State: Initialized, Role: FOLLOWER
I20260924 15:28:10.350517 13742 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:10.350594 13742 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:10.350622 13742 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:10.350669 13742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:10.351236 13742 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:10.351329 13742 leader_election.cc:302] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7513f8505fb409fbe5083b9e16b851e; no voters:
I20260924 15:28:10.351583 13742 leader_election.cc:288] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:10.351608 13747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:10.351785 13747 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 LEADER]: Becoming Leader. State: Replica: d7513f8505fb409fbe5083b9e16b851e, State: Running, Role: LEADER
I20260924 15:28:10.351909 13742 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:10.351934 13747 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:10.352387 13747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7513f8505fb409fbe5083b9e16b851e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } } }
I20260924 15:28:10.352644 13747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:10.352828 13747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7513f8505fb409fbe5083b9e16b851e. Latest consensus state: current_term: 1 leader_uuid: "d7513f8505fb409fbe5083b9e16b851e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } } }
I20260924 15:28:10.352973 13747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:10.353037 13756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:10.353645 13756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:10.355362 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.345150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.5.98.254:41601)
0924 15:28:10.345317 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:10.345326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:10.345350 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:28:10.345766 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:10.345772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:10.345816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:10.345929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:10.346708 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.347713 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.348742 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.349163 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.349243 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.351877 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:10.351881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:10.351881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:10.351893 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:10.353886 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.354205 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.354209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.354209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.354317 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.354568 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.354569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.354570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.354797 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:10.354950 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:10.355058 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:28:10.355209 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 15:28:10.361537 13756 catalog_manager.cc:1347] Generated new cluster ID: d1e1ecd90188484580d1d0b9dd28369c
I20260924 15:28:10.361584 13756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:28:10.387741 13756 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:10.388283 13756 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:10.402920 13756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Generated new TSK 0
I20260924 15:28:10.403132 13756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:28:10.434082 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:10.532428 13768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:10.532651 13768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:10.532678 13768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:10.533941 13768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:10.533987 13768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:10.534016 13768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:10.535564 13768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:10.535858 13768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:10.536113 13768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:10.541435 13773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.541455 13774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:10.541599 13768 server_base.cc:1034] running on GCE node
W20260924 15:28:10.541505 13776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:10.541864 13768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:10.542057 13768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:10.543205 13768 hybrid_clock.cc:648] HybridClock initialized: now 1790263690543184 us; error 41 us; skew 500 ppm
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:10.545674 13768 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:28:10.546139 13768 webserver.cc:460] Webserver started at http://127.5.98.193:37137/ using document root <none> and password file <none>
I20260924 15:28:10.546346 13768 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:10.546409 13768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:10.546538 13768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:10.547551 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "87ad6c454b9448c7ba603b68cc8dc3d1"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "f7f8564042ad0d3f574cf6f06b188441"
server_key_iv: "d5ea4c2a0058eabe5eb93b68d2827715"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.547973 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "87ad6c454b9448c7ba603b68cc8dc3d1"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "f7f8564042ad0d3f574cf6f06b188441"
server_key_iv: "d5ea4c2a0058eabe5eb93b68d2827715"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.549393 13768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:10.550156 13783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.550339 13768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:10.550454 13768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "87ad6c454b9448c7ba603b68cc8dc3d1"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "f7f8564042ad0d3f574cf6f06b188441"
server_key_iv: "d5ea4c2a0058eabe5eb93b68d2827715"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.550529 13768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:10.581887 13768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:10.583861 13768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:10.584045 13768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:10.584307 13768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:10.584638 13768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:10.584693 13768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.584739 13768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:10.584764 13768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.590703 13768 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:39661
I20260924 15:28:10.590757 13896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:39661 every 8 connection(s)
I20260924 15:28:10.591100 13768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:10.600147 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:60773 (local address 127.5.98.254:41601)
0924 15:28:10.592000 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:10.592002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:10.592154 (+ 152us) server_negotiation.cc:407] Connection header received
0924 15:28:10.592679 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:10.592680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:10.592691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:10.592760 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:10.593669 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.594188 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.595177 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.595392 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.595455 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.597501 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:10.597503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:10.597503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:10.597510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:10.598886 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.599225 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.599226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.599226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.599276 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.599482 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.599482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.599483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.599587 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:10.599636 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:10.599923 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 15:28:10.600035 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:28:10.600165 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13768
I20260924 15:28:10.600167 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.591875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:60773)
0924 15:28:10.592027 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:10.592068 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:10.592552 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:10.592815 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:10.592819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:10.593374 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:10.593565 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.593569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.594277 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.594277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.595066 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.595068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.595470 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.595470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.595571 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.596104 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:10.596112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:10.597418 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:10.598963 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.598967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.598969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.599164 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.599333 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.599334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.599334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.599428 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.599642 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:10.599647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:10.599812 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 15:28:10.599895 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:28:10.600008 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260924 15:28:10.600270 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:10.600466 5515 external_mini_cluster.cc:1428] Setting key ddd27c6a688727157d66dcda4132ae6b
I20260924 15:28:10.600920 13897 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41601
I20260924 15:28:10.601070 13897 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:10.601284 13897 heartbeater.cc:511] Master 127.5.98.254:41601 requested a full tablet report, sending...
I20260924 15:28:10.601874 13706 ts_manager.cc:194] Registered new tserver with Master: 87ad6c454b9448c7ba603b68cc8dc3d1 (127.5.98.193:39661)
I20260924 15:28:10.602720 13706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:60773
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:28:10.624532 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:10.706782 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:10.707010 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:10.707047 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:10.708240 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:10.708305 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:10.708351 13902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:10.709780 13902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:10.710062 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:10.710307 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:10.715366 13910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.715337 13908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.715344 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:10.715626 13902 server_base.cc:1034] running on GCE node
I20260924 15:28:10.715864 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:10.716053 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:10.717221 13902 hybrid_clock.cc:648] HybridClock initialized: now 1790263690717212 us; error 41 us; skew 500 ppm
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:10.719725 13902 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:28:10.720188 13902 webserver.cc:460] Webserver started at http://127.5.98.194:38557/ using document root <none> and password file <none>
I20260924 15:28:10.720408 13902 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:10.720472 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:10.720628 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:10.721662 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "9c26020e04c3428895835d1f76673342"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "bb4972b0ec02b84d1530b53a323687c4"
server_key_iv: "26e773e287e4500075b2cfbf8f6f4b6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.722028 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "9c26020e04c3428895835d1f76673342"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "bb4972b0ec02b84d1530b53a323687c4"
server_key_iv: "26e773e287e4500075b2cfbf8f6f4b6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.723340 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:28:10.724159 13917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.724315 13902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.724414 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "9c26020e04c3428895835d1f76673342"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "bb4972b0ec02b84d1530b53a323687c4"
server_key_iv: "26e773e287e4500075b2cfbf8f6f4b6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.724494 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:10.732785 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:10.735620 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:10.735808 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:10.736064 13902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:10.736429 13902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:10.736696 13902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.736737 13902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:10.736765 13902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.745452 13902 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:43273
I20260924 15:28:10.745530 14030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:43273 every 8 connection(s)
I20260924 15:28:10.745929 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:28:10.749058 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 13902
I20260924 15:28:10.749190 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:10.749408 5515 external_mini_cluster.cc:1428] Setting key 9163589ac62892673f1a9f10181cadee
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:28:10.760870 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.746998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:33125 (local address 127.5.98.254:41601)
0924 15:28:10.747051 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:10.747053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:10.747065 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:10.747935 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:10.747937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:10.747952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:10.748016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:10.749286 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.750041 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.751642 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.751999 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.752077 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.757635 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:10.757637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:10.757637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:10.757646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:10.759150 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.759743 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.759744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.759745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.759813 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.760154 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.760155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.760156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.760298 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:10.760362 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:10.760689 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 15:28:10.760779 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:28:10.761061 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.746690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:33125)
0924 15:28:10.746838 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:10.746914 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:10.747814 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:10.748091 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:10.748095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:10.748851 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:10.749160 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.749167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.750158 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.750158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.751543 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.751547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.752112 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.752114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.752298 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.755460 (+ 3162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:10.755472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:10.757534 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:10.759259 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.759268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.759271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.759599 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.759923 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.759923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.759924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.760074 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.760398 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:10.760405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:10.760529 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 15:28:10.760642 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:28:10.760810 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260924 15:28:10.761641 14031 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41601
I20260924 15:28:10.761763 14031 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:10.761999 14031 heartbeater.cc:511] Master 127.5.98.254:41601 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
I20260924 15:28:10.762408 13706 ts_manager.cc:194] Registered new tserver with Master: 9c26020e04c3428895835d1f76673342 (127.5.98.194:43273)
I20260924 15:28:10.762980 13706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:33125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:28:10.777817 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:10.868654 14036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:10.868911 14036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:10.868947 14036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:10.870783 14036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:10.870864 14036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:10.870923 14036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:10.873467 14036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:10.873749 14036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:10.874049 14036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:10.881227 14042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.881253 14041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:10.881294 14044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:10.881469 14036 server_base.cc:1034] running on GCE node
I20260924 15:28:10.881740 14036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:10.881997 14036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:10.883184 14036 hybrid_clock.cc:648] HybridClock initialized: now 1790263690883169 us; error 32 us; skew 500 ppm
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:10.885727 14036 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:28:10.886176 14036 webserver.cc:460] Webserver started at http://127.5.98.195:43383/ using document root <none> and password file <none>
I20260924 15:28:10.886423 14036 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:10.886492 14036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:10.886641 14036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:10.887804 14036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "de8293d888d0473cb167c7360fe3af86"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "c1d4c5616973beae9aab6274d5cc1057"
server_key_iv: "f994d9bb72d77b3ef285a97527647722"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.888167 14036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "de8293d888d0473cb167c7360fe3af86"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "c1d4c5616973beae9aab6274d5cc1057"
server_key_iv: "f994d9bb72d77b3ef285a97527647722"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.889545 14036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:28:10.890379 14051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.890596 14036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:10.890709 14036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "de8293d888d0473cb167c7360fe3af86"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "c1d4c5616973beae9aab6274d5cc1057"
server_key_iv: "f994d9bb72d77b3ef285a97527647722"
server_key_version: "encryptionkey@0"
I20260924 15:28:10.890787 14036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:10.899287 14036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:10.901189 14036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:10.901342 14036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:10.901559 14036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:10.901853 14036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:10.901913 14036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.901954 14036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:10.901981 14036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:10.907805 14036 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:39875
I20260924 15:28:10.907846 14164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:39875 every 8 connection(s)
I20260924 15:28:10.908174 14036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:10.912325 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14036
I20260924 15:28:10.912425 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:10.912629 5515 external_mini_cluster.cc:1428] Setting key ebfeef4b43599484b081485effe63a7d
Sep 24 15:28:10 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263690, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:10.917294 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.908902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:42947 (local address 127.5.98.254:41601)
0924 15:28:10.908945 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:10.908947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:10.909145 (+ 198us) server_negotiation.cc:407] Connection header received
0924 15:28:10.909612 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:10.909613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:10.909622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:10.909684 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:10.910552 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.911033 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.912163 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.912473 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.912550 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.914730 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:10.914731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:10.914731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:10.914737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:10.916164 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.916525 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.916525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.916526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.916576 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:10.916784 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:10.916785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:10.916785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:10.916884 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:10.916942 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:10.917171 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 15:28:10.917231 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:10.917440 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:10.908870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:42947)
0924 15:28:10.909024 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:10.909064 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:10.909531 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:10.909737 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:10.909740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:10.910289 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:10.910477 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.910482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.911113 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.911113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.912067 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:10.912070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:10.912555 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:10.912556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:10.912655 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:10.913261 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:10.913271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:10.914651 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:10.916237 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.916243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.916246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.916458 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.916635 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:10.916636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:10.916636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:10.916730 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:10.916945 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:10.916950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:10.917054 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 15:28:10.917137 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:28:10.917252 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260924 15:28:10.917860 14165 heartbeater.cc:347] Connected to a master server at 127.5.98.254:41601
I20260924 15:28:10.917990 14165 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:10.918190 14165 heartbeater.cc:511] Master 127.5.98.254:41601 requested a full tablet report, sending...
I20260924 15:28:10.918569 13706 ts_manager.cc:194] Registered new tserver with Master: de8293d888d0473cb167c7360fe3af86 (127.5.98.195:39875)
I20260924 15:28:10.918946 13706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:42947
I20260924 15:28:10.924501 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
Entry for principal oryx/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab.
WARNING: no policy specified for oryx/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
Entry for principal oryx/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab.
WARNING: no policy specified for oryx/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
Entry for principal oryx/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab.
I20260924 15:28:10.990262 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13768
I20260924 15:28:10.995144 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13902
I20260924 15:28:10.999621 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14036
I20260924 15:28:11.004212 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 13674
I20260924 15:28:11.009135 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41601
--webserver_interface=127.5.98.254
--webserver_port=35023
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:41601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:11.091204 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:11.091410 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:11.091434 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:11.092577 14174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:11.092618 14174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:11.092633 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:11.092645 14174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:11.092657 14174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:11.094034 14174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:41601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:41601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:11.094236 14174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:11.094408 14174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:11.099334 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.099308 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.099308 14182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:11.099417 14174 server_base.cc:1034] running on GCE node
I20260924 15:28:11.099830 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:11.100076 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:11.101257 14174 hybrid_clock.cc:648] HybridClock initialized: now 1790263691101225 us; error 35 us; skew 500 ppm
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263691, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:11.103893 14174 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:28:11.104322 14174 webserver.cc:460] Webserver started at http://127.5.98.254:35023/ using document root <none> and password file <none>
I20260924 15:28:11.104547 14174 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:11.104602 14174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:11.105979 14174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:11.106612 14189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.106781 14174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:11.106879 14174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "d7513f8505fb409fbe5083b9e16b851e"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "0e9af8e77655ee3537cdab020bac1ee5"
server_key_iv: "f818b4cda4f539750b4b85e10f343e33"
server_key_version: "encryptionkey@0"
I20260924 15:28:11.107234 14174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:11.115823 14174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:11.117585 14174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:11.117750 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:11.121593 14174 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:41601
I20260924 15:28:11.121649 14241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:41601 every 8 connection(s)
I20260924 15:28:11.122035 14174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:11.123426 14242 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:28:11.123505 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14174
I20260924 15:28:11.123833 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:39661
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:41601
--webserver_port=37137
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:28:11.124220 14242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Bootstrap starting.
I20260924 15:28:11.132565 14242 log.cc:824] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:11.134336 14242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:28:11.134642 14242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Bootstrap complete.
I20260924 15:28:11.136914 14242 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:11.137144 14242 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7513f8505fb409fbe5083b9e16b851e, State: Initialized, Role: FOLLOWER
I20260924 15:28:11.137326 14242 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:11.137392 14242 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:11.137455 14242 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:11.137495 14242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:28:11.138492 14242 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:11.138581 14242 leader_election.cc:302] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7513f8505fb409fbe5083b9e16b851e; no voters:
I20260924 15:28:11.138805 14242 leader_election.cc:288] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:28:11.138849 14246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:28:11.138965 14246 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [term 2 LEADER]: Becoming Leader. State: Replica: d7513f8505fb409fbe5083b9e16b851e, State: Running, Role: LEADER
I20260924 15:28:11.139052 14246 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } }
I20260924 15:28:11.139079 14242 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:11.139302 14248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7513f8505fb409fbe5083b9e16b851e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } } }
I20260924 15:28:11.139392 14248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:11.139781 14247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7513f8505fb409fbe5083b9e16b851e. Latest consensus state: current_term: 2 leader_uuid: "d7513f8505fb409fbe5083b9e16b851e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7513f8505fb409fbe5083b9e16b851e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 41601 } } }
I20260924 15:28:11.139917 14247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:11.139940 14250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:11.140487 14250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:11.140650 14250 catalog_manager.cc:1259] Loaded cluster ID: d1e1ecd90188484580d1d0b9dd28369c
I20260924 15:28:11.140693 14250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:28:11.141546 14250 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:11.141742 14250 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7513f8505fb409fbe5083b9e16b851e: Loaded TSK: 0
I20260924 15:28:11.141957 14250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:28:11.218153 14244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:11.218401 14244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:11.218443 14244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:11.219627 14244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:11.219727 14244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:11.219776 14244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:11.221199 14244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.193.keytab
--principal=oryx/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:39661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=37137
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:11.221469 14244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:11.221706 14244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:11.226790 14268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.226783 14271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.226859 14269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:11.227104 14244 server_base.cc:1034] running on GCE node
I20260924 15:28:11.227344 14244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:11.227617 14244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:11.228790 14244 hybrid_clock.cc:648] HybridClock initialized: now 1790263691228786 us; error 33 us; skew 500 ppm
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263691, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:11.231292 14244 init.cc:377] Logged in from keytab as oryx/127.5.98.193@KRBTEST.COM (short username oryx)
I20260924 15:28:11.231820 14244 webserver.cc:460] Webserver started at http://127.5.98.193:37137/ using document root <none> and password file <none>
I20260924 15:28:11.232071 14244 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:11.232127 14244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:11.233525 14244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 15:28:11.234220 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.234390 14244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:11.234493 14244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "87ad6c454b9448c7ba603b68cc8dc3d1"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "f7f8564042ad0d3f574cf6f06b188441"
server_key_iv: "d5ea4c2a0058eabe5eb93b68d2827715"
server_key_version: "encryptionkey@0"
I20260924 15:28:11.234831 14244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:11.249588 14244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:11.251927 14244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:11.252079 14244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:11.252352 14244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:11.252672 14244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:11.252704 14244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.252727 14244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:11.252774 14244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.259997 14244 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:39661
I20260924 15:28:11.260102 14391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:39661 every 8 connection(s)
I20260924 15:28:11.260434 14244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:11.269629 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14244
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.269949 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:43273
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:41601
--webserver_port=38557
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.271831 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.261169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:57477)
0924 15:28:11.261370 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.261415 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.262016 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.264164 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.264169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.264904 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.265169 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.265175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.266246 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.266247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.267490 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.267494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.268141 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.268142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.268295 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.268815 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.268825 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.271591 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
W20260924 15:28:11.271872 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.261298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:57477 (local address 127.5.98.254:41601)
0924 15:28:11.263762 (+ 2464us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.263769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.263798 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:28:11.263904 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.263909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.263957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.264063 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.265279 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.266146 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.267578 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.268049 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.268129 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.271726 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:57477: BlockingRecv error: recv got EOF from 127.5.98.193:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":2420,"thread_start_us":70,"threads_started":1}
W20260924 15:28:11.272373 14392 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41601 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.98.254:41601: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.276883 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.272646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:35069)
0924 15:28:11.272739 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.272751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.272795 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.272939 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.272940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.273071 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.273112 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.273113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.273780 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.273782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.274314 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.274315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.274683 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.274684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.274751 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.275189 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.275194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.276805 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 15:28:11.276935 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.272696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:35069 (local address 127.5.98.254:41601)
0924 15:28:11.272745 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.272747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.272781 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:28:11.272852 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.272853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.272865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.272937 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.273186 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.273704 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.274410 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.274600 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.274682 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.276874 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:35069: BlockingRecv error: recv got EOF from 127.5.98.193:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.281261 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.277358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:51265)
0924 15:28:11.277400 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.277411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.277458 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.277591 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.277592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.277720 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.277744 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.277745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.278343 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.278344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.278833 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.278834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.279184 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.279185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.279236 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.279637 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.279641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.281187 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 15:28:11.281311 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.277411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:51265 (local address 127.5.98.254:41601)
0924 15:28:11.277454 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.277456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.277465 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:11.277518 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.277519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.277529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.277588 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.277828 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.278266 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.278923 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.279107 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.279173 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.281250 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:51265: BlockingRecv error: recv got EOF from 127.5.98.193:51265 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:28:11.281603 14392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:28:11.353438 14396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:11.353654 14396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:11.353677 14396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:11.354786 14396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:11.354822 14396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:11.354848 14396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:11.356220 14396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.194.keytab
--principal=oryx/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:43273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=38557
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:11.356406 14396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:11.356571 14396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:11.361668 14401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.361678 14404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.361696 14402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:11.361879 14396 server_base.cc:1034] running on GCE node
I20260924 15:28:11.362093 14396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:11.362305 14396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:11.363467 14396 hybrid_clock.cc:648] HybridClock initialized: now 1790263691363445 us; error 51 us; skew 500 ppm
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263691, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:11.365868 14396 init.cc:377] Logged in from keytab as oryx/127.5.98.194@KRBTEST.COM (short username oryx)
I20260924 15:28:11.366323 14396 webserver.cc:460] Webserver started at http://127.5.98.194:38557/ using document root <none> and password file <none>
I20260924 15:28:11.366561 14396 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:11.366649 14396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:11.368009 14396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:11.368629 14411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.368805 14396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:11.368909 14396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "9c26020e04c3428895835d1f76673342"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "bb4972b0ec02b84d1530b53a323687c4"
server_key_iv: "26e773e287e4500075b2cfbf8f6f4b6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:11.369242 14396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:11.380901 14396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:11.382647 14396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:11.382808 14396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:11.383029 14396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:11.383335 14396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:11.383387 14396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.383420 14396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:11.383447 14396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.389808 14396 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:43273
I20260924 15:28:11.389853 14524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:43273 every 8 connection(s)
I20260924 15:28:11.390213 14396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:28:11.395288 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14396
I20260924 15:28:11.395714 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:39875
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:41601
--webserver_port=43383
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:28:11.397614 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.390902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:44205 (local address 127.5.98.254:41601)
0924 15:28:11.390969 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.390971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.391212 (+ 241us) server_negotiation.cc:407] Connection header received
0924 15:28:11.391687 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.391688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.391726 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.391799 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.392695 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.393179 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.394093 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.394291 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.394357 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.397566 (+ 3209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:44205: BlockingRecv error: recv got EOF from 127.5.98.194:44205 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:28:11.397706 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.390908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:44205)
0924 15:28:11.391089 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.391123 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.391586 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.391857 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.391861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.392430 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.392611 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.392616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.393256 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.393257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.394003 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.394004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.394373 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.394374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.394480 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.394950 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.394959 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.397480 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
W20260924 15:28:11.398022 14525 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41601 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.98.254:41601: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.402395 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.398265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:59001)
0924 15:28:11.398315 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.398326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.398363 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.398502 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.398503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.398624 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.398666 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.398667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.399301 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.399302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.399891 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.399894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.400276 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.400277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.400347 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.400763 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.400768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.402318 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:28:11.402462 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.398326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59001 (local address 127.5.98.254:41601)
0924 15:28:11.398370 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.398372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.398382 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:11.398417 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.398418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.398439 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.398498 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.398743 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.399221 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.399987 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.400176 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.400263 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.402392 (+ 2129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:59001: BlockingRecv error: recv got EOF from 127.5.98.194:59001 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.406638 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.402744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:40291)
0924 15:28:11.402789 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.402798 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.402837 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.402967 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.402968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.403086 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.403109 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.403110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.403750 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.403751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.404227 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.404228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.404596 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.404596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.404646 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.405051 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.405056 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.406563 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:28:11.406690 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.402823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:40291 (local address 127.5.98.254:41601)
0924 15:28:11.402879 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.402881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.402889 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:11.402910 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.402911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.402920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.402966 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.403186 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.403646 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.404305 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.404524 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.404597 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.406629 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:40291: BlockingRecv error: recv got EOF from 127.5.98.194:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 15:28:11.406865 14525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:28:11.477421 14528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:11.477614 14528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:11.477639 14528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:11.478781 14528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:11.478824 14528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:11.478850 14528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:11.480326 14528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/krb5kdc/oryx_127.5.98.195.keytab
--principal=oryx/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:39875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=43383
--tserver_master_addrs=127.5.98.254:41601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:11.480556 14528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:11.480741 14528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:11.485523 14536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.485669 14534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:11.485531 14533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:11.485800 14528 server_base.cc:1034] running on GCE node
I20260924 15:28:11.486009 14528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:11.486207 14528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:11.487342 14528 hybrid_clock.cc:648] HybridClock initialized: now 1790263691487329 us; error 29 us; skew 500 ppm
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263691, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:11.489854 14528 init.cc:377] Logged in from keytab as oryx/127.5.98.195@KRBTEST.COM (short username oryx)
I20260924 15:28:11.490324 14528 webserver.cc:460] Webserver started at http://127.5.98.195:43383/ using document root <none> and password file <none>
I20260924 15:28:11.490602 14528 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:11.490671 14528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:11.492056 14528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:28:11.492662 14543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.492839 14528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:11.492941 14528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "de8293d888d0473cb167c7360fe3af86"
format_stamp: "Formatted at 2026-09-24 15:28:10 on dist-test-slave-4g6f"
server_key: "c1d4c5616973beae9aab6274d5cc1057"
server_key_iv: "f994d9bb72d77b3ef285a97527647722"
server_key_version: "encryptionkey@0"
I20260924 15:28:11.493295 14528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:11.515868 14528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:11.517863 14528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:11.518031 14528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:11.518278 14528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:11.518607 14528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:11.518662 14528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.518698 14528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:11.518725 14528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:11.524506 14528 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:39875
I20260924 15:28:11.524552 14656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:39875 every 8 connection(s)
I20260924 15:28:11.524924 14528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.531206 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14528
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.532505 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.525654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:51887)
0924 15:28:11.525810 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.525849 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.526341 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.526572 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.526575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.527166 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.527359 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.527364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.528112 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.528113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.528913 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.528915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.529283 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.529284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.529387 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.529952 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.529963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.532329 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260924 15:28:11.532505 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.525725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:51887 (local address 127.5.98.254:41601)
0924 15:28:11.525813 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.525815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.525921 (+ 106us) server_negotiation.cc:407] Connection header received
0924 15:28:11.526446 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.526448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.526458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.526513 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.527453 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.527998 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.529016 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.529203 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.529269 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.532435 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:51887: BlockingRecv error: recv got EOF from 127.5.98.195:51887 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 15:28:11.532804 14657 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:41601 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.98.254:41601: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.537688 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.533052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:54047)
0924 15:28:11.533143 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.533154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.533204 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.533326 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.533326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.533464 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.533518 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.533519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.534357 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.534358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.535091 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.535093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.535452 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.535453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.535520 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.535914 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.535918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.537644 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 15:28:11.537804 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.533100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:54047 (local address 127.5.98.254:41601)
0924 15:28:11.533153 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.533154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.533190 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:28:11.533256 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.533257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.533275 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.533313 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.533582 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.534282 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.535181 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.535370 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.535440 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.537747 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:54047: BlockingRecv error: recv got EOF from 127.5.98.195:54047 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:28:11.540033 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.531781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.5.98.254:41601)
0924 15:28:11.531920 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.531921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.531931 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:11.531954 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.531956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.531968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.532046 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.532740 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.533478 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.534550 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.534850 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.534929 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.536271 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:11.536275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:11.536276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:11.536288 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:11.538227 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:11.538780 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:11.538784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:11.538784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:11.538933 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:11.539136 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:11.539137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:11.539137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:11.539419 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:11.539563 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:11.539784 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:28:11.539932 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:11 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:11.542868 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:11.538273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:57845)
0924 15:28:11.538316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:11.538326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:11.538388 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:11.538525 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:11.538526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:11.538663 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:11.538697 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.538699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.539509 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.539510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.540383 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:11.540385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.540839 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.540839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:11.540914 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.541292 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:11.541297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:11.542825 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:28:11.543056 14657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:28:11.543109 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:11.538318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:57845 (local address 127.5.98.254:41601)
0924 15:28:11.538352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:11.538354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:11.538371 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:28:11.538455 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:11.538455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:11.538462 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:11.538502 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:11.538771 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.539439 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.540452 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:11.540760 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:11.540814 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:11.543059 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:57845: BlockingRecv error: recv got EOF from 127.5.98.195:57845 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:12.286671 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:12.282025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:55499)
0924 15:28:12.282154 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:12.282173 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:12.282224 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:12.282514 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:12.282515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:12.282696 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:12.282748 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.282750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.283400 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.283401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.284017 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.284019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.284351 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.284351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.284403 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.284922 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:12.284927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:12.286604 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
W20260924 15:28:12.286773 14663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:12.282205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:55499 (local address 127.5.98.254:41601)
0924 15:28:12.282340 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:12.282342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:12.282349 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:12.282371 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:12.282372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:12.282382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:12.282447 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:12.282814 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.283311 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.284084 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.284270 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.284321 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.286699 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:55499: BlockingRecv error: recv got EOF from 127.5.98.193:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:12.412147 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:12.407279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:51757)
0924 15:28:12.407471 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:12.407487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:12.407565 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:12.407746 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:12.407747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:12.407927 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:12.407974 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.407976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.408714 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.408715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.409276 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.409277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.409647 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.409648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.409712 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.410288 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:12.410292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:12.412041 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":62,"threads_started":1}
W20260924 15:28:12.412184 14663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:12.407473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:51757 (local address 127.5.98.254:41601)
0924 15:28:12.407541 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:12.407543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:12.407553 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:12.407627 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:12.407628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:12.407643 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:12.407738 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:12.408056 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.408633 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.409361 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.409556 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.409647 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.412122 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:51757: BlockingRecv error: recv got EOF from 127.5.98.194:51757 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:12 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:12.553135 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:12.543428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:39059)
0924 15:28:12.543577 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:12.543588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:12.543646 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:12.543791 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:12.543792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:12.543969 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:12.544039 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.544040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.544655 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.544656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.545225 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:12.545227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.545528 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.545528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:12.545566 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.551122 (+ 5556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:12.551128 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:12.552994 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
W20260924 15:28:12.553179 14663 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:12.543520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:39059 (local address 127.5.98.254:41601)
0924 15:28:12.543585 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:12.543588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:12.543630 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:28:12.543717 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:12.543718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:12.543730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:12.543769 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:12.544111 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.544589 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.545296 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:12.545468 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:12.545515 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:12.553105 (+ 7590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:39059: BlockingRecv error: recv got EOF from 127.5.98.195:39059 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:13.291703 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:13.287171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:33333)
0924 15:28:13.287318 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:13.287329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:13.287390 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:13.287525 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:13.287526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:13.287751 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:13.287814 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.287816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.288486 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.288488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.289016 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.289018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.289457 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.289458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.289516 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.289932 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:13.289936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:13.291591 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260924 15:28:13.291793 14667 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:13.287264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:33333 (local address 127.5.98.254:41601)
0924 15:28:13.287366 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:13.287368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:13.287380 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:28:13.287441 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:13.287442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:13.287454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:13.287511 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:13.287889 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.288400 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.289157 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.289354 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.289458 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.291721 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:33333: BlockingRecv error: recv got EOF from 127.5.98.193:33333 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:13.417137 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:13.412706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:35337)
0924 15:28:13.412876 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:13.412887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:13.412954 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:13.413079 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:13.413080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:13.413232 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:13.413283 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.413285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.413909 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.413910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.414464 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.414466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.414775 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.414776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.414814 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.415325 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:13.415330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:13.417071 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
W20260924 15:28:13.417289 14667 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:13.412809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:35337 (local address 127.5.98.254:41601)
0924 15:28:13.412873 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:13.412875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:13.412945 (+ 70us) server_negotiation.cc:407] Connection header received
0924 15:28:13.413007 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:13.413008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:13.413019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:13.413071 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:13.413336 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.413835 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.414532 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.414714 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.414763 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.417207 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:35337: BlockingRecv error: recv got EOF from 127.5.98.194:35337 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:13 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:13.558224 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:13.553653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:56973)
0924 15:28:13.553817 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:13.553829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:13.553888 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:13.554022 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:13.554022 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:13.554198 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:13.554248 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.554250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.554894 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.554895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.555430 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:13.555432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.555793 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.555794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:13.555834 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.556349 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:13.556353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:13.558155 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
W20260924 15:28:13.558326 14667 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:13.553828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:56973 (local address 127.5.98.254:41601)
0924 15:28:13.553901 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:13.553903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:13.553910 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:13.553936 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:13.553937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:13.553949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:13.554000 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:13.554316 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.554821 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.555514 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:13.555730 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:13.555785 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:13.558255 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:56973: BlockingRecv error: recv got EOF from 127.5.98.195:56973 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:14.297537 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:14.292197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:43039)
0924 15:28:14.292323 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:14.292342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:14.292394 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:14.292696 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:14.292698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:14.292967 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:14.293038 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.293041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.293809 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.293810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.294401 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.294403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.294724 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.294724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.294768 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.295336 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:14.295340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:14.297440 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260924 15:28:14.297596 14671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:14.292304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:43039 (local address 127.5.98.254:41601)
0924 15:28:14.292426 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:14.292428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:14.292438 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:14.292473 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:14.292474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:14.292485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:14.292605 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:14.293170 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.293735 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.294471 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.294656 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.294706 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.297523 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:43039: BlockingRecv error: recv got EOF from 127.5.98.193:43039 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:14.422330 14672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:14.417632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:34435)
0924 15:28:14.417776 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:14.417787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:14.417847 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:14.418006 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:14.418007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:14.418171 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:14.418223 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.418225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.418923 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.418924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.419463 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.419465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.419887 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.419888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.419942 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.420480 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:14.420486 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:14.422269 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
W20260924 15:28:14.422494 14671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:14.417749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:34435 (local address 127.5.98.254:41601)
0924 15:28:14.417797 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:14.417800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:14.417832 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:28:14.417922 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:14.417923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:14.417935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:14.417981 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:14.418328 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.418831 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.419547 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.419817 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.419876 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.422414 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:34435: BlockingRecv error: recv got EOF from 127.5.98.194:34435 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:14 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:14.563707 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:14.558758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:54775)
0924 15:28:14.559050 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:14.559063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:14.559160 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:14.559341 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:14.559343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:14.559536 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:14.559587 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.559589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.560333 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.560335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.560889 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:14.560891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.561296 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.561297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:14.561367 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.561817 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:14.561821 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:14.563590 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
W20260924 15:28:14.563764 14671 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:14.558843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:54775 (local address 127.5.98.254:41601)
0924 15:28:14.559032 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:14.559034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:14.559141 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:28:14.559250 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:14.559251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:14.559266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:14.559331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:14.559693 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.560228 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.560987 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:14.561184 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:14.561283 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:14.563702 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:54775: BlockingRecv error: recv got EOF from 127.5.98.195:54775 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:15.302922 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:15.298170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:54123)
0924 15:28:15.298330 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:15.298343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:15.298410 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:15.298603 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:15.298603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:15.298787 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:15.298838 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.298840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.299524 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.299525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.300071 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.300072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.300454 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.300455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.300494 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.300987 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:15.300991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:15.302864 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
W20260924 15:28:15.303054 14675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:15.298239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:54123 (local address 127.5.98.254:41601)
0924 15:28:15.298468 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:15.298469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:15.298479 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:15.298503 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:15.298504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:15.298515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:15.298584 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:15.298911 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.299452 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.300150 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.300348 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.300448 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.302966 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:54123: BlockingRecv error: recv got EOF from 127.5.98.193:54123 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:15.427757 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:15.422968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:33595)
0924 15:28:15.423105 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:15.423125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:15.423176 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:15.423332 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:15.423333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:15.423500 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:15.423550 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.423551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.424388 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.424389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.425025 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.425028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.425386 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.425387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.425432 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.425871 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:15.425875 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:15.427631 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260924 15:28:15.427839 14675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:15.423055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:33595 (local address 127.5.98.254:41601)
0924 15:28:15.423147 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:15.423149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:15.423163 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:28:15.423236 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:15.423237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:15.423248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:15.423308 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:15.423610 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.424257 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.425115 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.425308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.425373 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.427777 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:33595: BlockingRecv error: recv got EOF from 127.5.98.194:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:15 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:15.568938 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:15.564185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:32947)
0924 15:28:15.564373 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:15.564385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:15.564481 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:15.564626 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:15.564627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:15.564816 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:15.564864 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.564866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.565578 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.565579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.566130 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:15.566132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.566523 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.566524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:15.566593 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.567048 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:15.567052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:15.568838 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
W20260924 15:28:15.569003 14675 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:15.564244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:32947 (local address 127.5.98.254:41601)
0924 15:28:15.564298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:15.564314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:15.564475 (+ 161us) server_negotiation.cc:407] Connection header received
0924 15:28:15.564538 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:15.564539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:15.564555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:15.564611 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:15.564961 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.565487 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.566247 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:15.566448 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:15.566525 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:15.568923 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:32947: BlockingRecv error: recv got EOF from 127.5.98.195:32947 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:16.307878 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:16.303409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:47863)
0924 15:28:16.303574 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:16.303584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:16.303632 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:16.303823 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:16.303824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:16.303984 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:16.304054 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.304056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.304676 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.304677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.305198 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.305200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.305512 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.305513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.305556 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.306030 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:16.306034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:16.307813 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
W20260924 15:28:16.308027 14679 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:16.303470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:47863 (local address 127.5.98.254:41601)
0924 15:28:16.303593 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:16.303599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:16.303620 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:28:16.303730 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:16.303731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:16.303747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:16.303806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:16.304117 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.304612 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.305265 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.305451 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.305498 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.307908 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:47863: BlockingRecv error: recv got EOF from 127.5.98.193:47863 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:16.432761 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:16.428253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:48659)
0924 15:28:16.428395 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:16.428406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:16.428458 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:16.428698 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:16.428699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:16.428845 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:16.428896 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.428897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.429534 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.429535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.430056 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.430057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.430438 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.430439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.430506 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.430922 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:16.430926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:16.432665 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
W20260924 15:28:16.432812 14679 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:16.428504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:48659 (local address 127.5.98.254:41601)
0924 15:28:16.428575 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:16.428577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:16.428584 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:16.428610 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:16.428610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:16.428620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:16.428669 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:16.428959 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.429468 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.430131 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.430357 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.430429 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.432749 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:48659: BlockingRecv error: recv got EOF from 127.5.98.194:48659 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:16 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:16.574370 14681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:16.569451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:47877)
0924 15:28:16.569589 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:16.569600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:16.569664 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:16.569805 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:16.569806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:16.570045 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:16.570128 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.570131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.571120 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.571121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.571685 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:16.571687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.572009 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.572010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:16.572052 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.572551 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:16.572556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:16.574312 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260924 15:28:16.574477 14679 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:16.569515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:47877 (local address 127.5.98.254:41601)
0924 15:28:16.569593 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:16.569596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:16.569667 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:28:16.569710 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:16.569711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:16.569724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:16.569794 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:16.570224 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.571027 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.571757 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:16.571939 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:16.572001 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:16.574405 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:47877: BlockingRecv error: recv got EOF from 127.5.98.195:47877 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:17.312872 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:17.308337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:53923)
0924 15:28:17.308481 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:17.308491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:17.308542 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:17.308685 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:17.308686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:17.308841 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:17.308889 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.308890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.309519 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.309520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.310037 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.310038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.310374 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.310375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.310431 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.310949 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:17.310953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:17.312801 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
W20260924 15:28:17.313046 14683 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:17.308383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:53923 (local address 127.5.98.254:41601)
0924 15:28:17.308555 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:17.308557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:17.308584 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:28:17.308605 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:17.308606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:17.308616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:17.308667 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:17.308951 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.309454 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.310123 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.310324 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.310373 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.312957 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:53923: BlockingRecv error: recv got EOF from 127.5.98.193:53923 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:17.438020 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:17.433263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:53461)
0924 15:28:17.433404 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:17.433417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:17.433489 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:17.433623 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:17.433623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:17.433776 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:17.433827 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.433828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.434574 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.434576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.435289 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.435292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.435612 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.435613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.435651 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.436077 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:17.436081 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:17.437963 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260924 15:28:17.438088 14683 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:17.433341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:53461 (local address 127.5.98.254:41601)
0924 15:28:17.433417 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:17.433419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:17.433489 (+ 70us) server_negotiation.cc:407] Connection header received
0924 15:28:17.433540 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:17.433540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:17.433552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:17.433610 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:17.433886 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.434448 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.435359 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.435546 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.435605 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.438040 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:53461: BlockingRecv error: recv got EOF from 127.5.98.194:53461 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:17 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:17.582986 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:17.574851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:36107)
0924 15:28:17.575011 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:17.575021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:17.575079 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:17.575232 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:17.575233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:17.575406 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:17.575456 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.575457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.576366 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.576367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.576916 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:17.576918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.577231 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.577232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:17.577274 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.581150 (+ 3876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:17.581155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:17.582928 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
W20260924 15:28:17.583110 14683 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:17.574921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:36107 (local address 127.5.98.254:41601)
0924 15:28:17.574985 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:17.574987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:17.575072 (+ 85us) server_negotiation.cc:407] Connection header received
0924 15:28:17.575144 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:17.575144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:17.575157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:17.575216 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:17.575547 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.576295 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.576991 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:17.577169 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:17.577218 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:17.583029 (+ 5811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:36107: BlockingRecv error: recv got EOF from 127.5.98.195:36107 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:18.317740 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:18.313284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:51721)
0924 15:28:18.313407 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:18.313420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:18.313482 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:18.313599 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:18.313599 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:18.313755 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:18.313805 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.313807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.314499 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.314500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.315019 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.315020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.315406 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.315406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.315448 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.315978 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:18.315982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:18.317682 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
W20260924 15:28:18.317909 14687 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:18.313340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:51721 (local address 127.5.98.254:41601)
0924 15:28:18.313486 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:18.313488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:18.313498 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:18.313522 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:18.313523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:18.313533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:18.313595 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:18.313863 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.314430 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.315090 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.315333 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.315402 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.317792 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:51721: BlockingRecv error: recv got EOF from 127.5.98.193:51721 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:18.443493 14688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:18.438552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:45425)
0924 15:28:18.438757 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:18.438766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:18.438820 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:18.438975 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:18.438975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:18.439188 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:18.439240 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.439242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.440040 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.440042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.440616 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.440618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.441002 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.441002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.441044 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.441588 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:18.441594 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:18.443435 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":135,"threads_started":1}
W20260924 15:28:18.443722 14687 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:18.438596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:45425 (local address 127.5.98.254:41601)
0924 15:28:18.438675 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:18.438678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:18.438816 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:28:18.438899 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:18.438900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:18.438925 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:18.438971 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:18.439297 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.439964 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.440709 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.440916 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.440986 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.443604 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:45425: BlockingRecv error: recv got EOF from 127.5.98.194:45425 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:18 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:18.588088 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:18.583483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:52965)
0924 15:28:18.583632 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:18.583646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:18.583749 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:18.583874 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:18.583875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:18.584055 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:18.584103 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.584104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.584810 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.584811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.585354 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:18.585356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.585689 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.585690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:18.585738 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.586255 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:18.586260 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:18.588030 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260924 15:28:18.588239 14687 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:18.583553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:52965 (local address 127.5.98.254:41601)
0924 15:28:18.583639 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:18.583641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:18.583748 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:28:18.583808 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:18.583809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:18.583821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:18.583860 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:18.584199 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.584720 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.585429 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:18.585620 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:18.585674 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:18.588156 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:52965: BlockingRecv error: recv got EOF from 127.5.98.195:52965 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:19.323006 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:19.318391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:37475)
0924 15:28:19.318561 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:19.318573 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:19.318625 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:19.318794 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:19.318794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:19.318955 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:19.319008 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.319009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.319702 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.319703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.320368 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.320370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.320725 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.320726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.320766 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.321201 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:19.321206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:19.322929 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
W20260924 15:28:19.323145 14691 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:19.318516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:37475 (local address 127.5.98.254:41601)
0924 15:28:19.318666 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:19.318668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:19.318675 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:19.318699 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:19.318700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:19.318710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:19.318774 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:19.319075 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.319570 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.320441 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.320662 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.320717 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.323023 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:37475: BlockingRecv error: recv got EOF from 127.5.98.193:37475 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:19.448531 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:19.443946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:42265)
0924 15:28:19.444101 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:19.444119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:19.444175 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:19.444296 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:19.444296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:19.444450 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:19.444502 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.444503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.445274 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.445275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.445800 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.445801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.446122 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.446122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.446162 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.446672 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:19.446677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:19.448463 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
W20260924 15:28:19.448670 14691 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:19.444005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:42265 (local address 127.5.98.254:41601)
0924 15:28:19.444094 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:19.444096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:19.444176 (+ 80us) server_negotiation.cc:407] Connection header received
0924 15:28:19.444233 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:19.444234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:19.444247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:19.444292 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:19.444561 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.445204 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.445869 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.446059 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.446111 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.448588 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:42265: BlockingRecv error: recv got EOF from 127.5.98.194:42265 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:19 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:19.593425 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:19.588568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:59257)
0924 15:28:19.588731 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:19.588741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:19.588796 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:19.588914 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:19.588915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:19.589085 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:19.589161 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.589163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.590033 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.590034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.590576 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:19.590577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.590917 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.590918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:19.590966 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.591545 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:19.591550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:19.593355 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
W20260924 15:28:19.593571 14691 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:19.588641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:59257 (local address 127.5.98.254:41601)
0924 15:28:19.588728 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:19.588731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:19.588798 (+ 67us) server_negotiation.cc:407] Connection header received
0924 15:28:19.588854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:19.588855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:19.588868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:19.588909 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:19.589245 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.589960 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.590662 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:19.590846 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:19.590903 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:19.593456 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:59257: BlockingRecv error: recv got EOF from 127.5.98.195:59257 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:20.327848 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:20.323469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:51207)
0924 15:28:20.323706 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:20.323718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:20.323787 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:20.323911 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:20.323912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:20.324067 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:20.324117 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.324119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.324737 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.324739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.325245 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.325246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.325553 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.325554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.325600 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.326043 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:20.326047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:20.327781 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
W20260924 15:28:20.327955 14695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:20.323584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:51207 (local address 127.5.98.254:41601)
0924 15:28:20.323734 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:20.323736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:20.323783 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:28:20.323829 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:20.323830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:20.323839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:20.323896 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:20.324177 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.324672 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.325307 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.325492 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.325540 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.327879 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:51207: BlockingRecv error: recv got EOF from 127.5.98.193:51207 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:20.453568 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:20.449022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:58217)
0924 15:28:20.449142 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:20.449160 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:20.449219 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:20.449346 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:20.449347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:20.449497 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:20.449553 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.449555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.450191 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.450192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.450722 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.450723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.451114 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.451115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.451181 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.451715 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:20.451720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:20.453467 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260924 15:28:20.453625 14695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:20.449084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:58217 (local address 127.5.98.254:41601)
0924 15:28:20.449140 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:20.449142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:20.449220 (+ 78us) server_negotiation.cc:407] Connection header received
0924 15:28:20.449265 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:20.449266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:20.449278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:20.449331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:20.449615 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.450115 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.450820 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.451026 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.451112 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.453562 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:58217: BlockingRecv error: recv got EOF from 127.5.98.194:58217 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:20 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:20.598398 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:20.593977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:40699)
0924 15:28:20.594106 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:20.594118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:20.594178 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:20.594311 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:20.594311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:20.594466 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:20.594515 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.594517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.595136 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.595137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.595773 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:20.595776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.596095 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.596096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:20.596133 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.596619 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:20.596624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:20.598342 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260924 15:28:20.598556 14695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:20.594112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:40699 (local address 127.5.98.254:41601)
0924 15:28:20.594201 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:20.594203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:20.594217 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:28:20.594239 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:20.594240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:20.594253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:20.594306 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:20.594577 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.595066 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.595841 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:20.596028 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:20.596084 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:20.598465 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:40699: BlockingRecv error: recv got EOF from 127.5.98.195:40699 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:21.332657 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:21.328326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:42479)
0924 15:28:21.328574 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:21.328594 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:21.328648 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:21.328785 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:21.328785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:21.328935 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:21.328983 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.328984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.329618 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.329619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.330123 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.330124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.330444 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.330445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.330483 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.330886 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:21.330890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:21.332600 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":202,"threads_started":1}
W20260924 15:28:21.332788 14699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:21.328526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:42479 (local address 127.5.98.254:41601)
0924 15:28:21.328631 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:21.328632 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:21.328640 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:21.328709 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:21.328709 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:21.328720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:21.328772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:21.329041 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.329549 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.330191 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.330382 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.330431 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.332683 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:42479: BlockingRecv error: recv got EOF from 127.5.98.193:42479 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:21.458863 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:21.454097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:39379)
0924 15:28:21.454253 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:21.454265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:21.454329 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:21.454497 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:21.454498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:21.454658 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:21.454709 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.454710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.455352 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.455353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.455987 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.455989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.456421 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.456422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.456476 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.457009 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:21.457013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:21.458768 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
W20260924 15:28:21.458916 14699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:21.454162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:39379 (local address 127.5.98.254:41601)
0924 15:28:21.454236 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:21.454238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:21.454312 (+ 74us) server_negotiation.cc:407] Connection header received
0924 15:28:21.454388 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:21.454389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:21.454399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:21.454441 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:21.454783 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.455283 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.456086 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.456296 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.456351 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.458854 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:39379: BlockingRecv error: recv got EOF from 127.5.98.194:39379 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:21 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:21.603233 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:21.598803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:38135)
0924 15:28:21.598949 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:21.598959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:21.598999 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:21.599120 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:21.599121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:21.599287 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:21.599335 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.599337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.600109 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.600111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.600651 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:21.600653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.600979 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.600980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:21.601025 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.601477 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:21.601482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:21.603175 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260924 15:28:21.603366 14699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:21.598927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:38135 (local address 127.5.98.254:41601)
0924 15:28:21.598980 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:21.598982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:21.599005 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:28:21.599047 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:21.599047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:21.599059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:21.599099 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:21.599401 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.599958 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.600732 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:21.600912 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:21.600964 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:21.603281 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:38135: BlockingRecv error: recv got EOF from 127.5.98.195:38135 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:22.337826 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:22.333116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:60895)
0924 15:28:22.333263 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:22.333273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:22.333326 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:22.333495 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:22.333496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:22.333659 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:22.333707 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.333709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.334362 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.334363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.334888 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.334890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.335232 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.335233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.335273 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.335815 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:22.335820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:22.337719 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260924 15:28:22.337867 14703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:22.333204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:60895 (local address 127.5.98.254:41601)
0924 15:28:22.333359 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:22.333361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:22.333370 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:22.333395 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:22.333395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:22.333413 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:22.333477 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:22.333774 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.334296 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.334956 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.335166 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.335219 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.337796 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:60895: BlockingRecv error: recv got EOF from 127.5.98.193:60895 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:22.464037 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:22.459366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:59091)
0924 15:28:22.459574 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:22.459586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:22.459696 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:22.459867 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:22.459868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:22.460051 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:22.460097 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.460099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.460771 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.460773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.461318 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.461320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.461693 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.461694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.461763 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.462197 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:22.462201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:22.463931 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
W20260924 15:28:22.464088 14703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:22.459460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59091 (local address 127.5.98.254:41601)
0924 15:28:22.459550 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:22.459552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:22.459676 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:28:22.459771 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:22.459772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:22.459784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:22.459857 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:22.460178 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.460689 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.461406 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.461607 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.461693 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.464022 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:59091: BlockingRecv error: recv got EOF from 127.5.98.194:59091 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:22 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:22.615337 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:22.603816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:57289)
0924 15:28:22.603960 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:22.603978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:22.604035 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:22.604191 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:22.604192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:22.604406 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:22.604492 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.604495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.605291 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.605292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.605858 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:22.605860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.606179 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.606180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:22.606229 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.613424 (+ 7195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:22.613429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:22.615229 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260924 15:28:22.615370 14703 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:22.603888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:57289 (local address 127.5.98.254:41601)
0924 15:28:22.603957 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:22.603963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:22.604036 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:28:22.604095 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:22.604095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:22.604107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:22.604176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:22.604580 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.605201 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.605936 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:22.606114 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:22.606163 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:22.615306 (+ 9143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:57289: BlockingRecv error: recv got EOF from 127.5.98.195:57289 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:23.343761 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:23.338325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:56981)
0924 15:28:23.338541 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:23.338552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:23.338607 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:23.338793 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:23.338793 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:23.339045 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:23.339097 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.339099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.339880 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.339881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.340504 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.340524 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.340913 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.340914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.340959 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.341543 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:23.341547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:23.343629 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":153,"threads_started":1}
W20260924 15:28:23.343881 14707 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:23.338452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:56981 (local address 127.5.98.254:41601)
0924 15:28:23.338589 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:23.338591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:23.338600 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:28:23.338687 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:23.338688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:23.338701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:23.338772 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:23.339179 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.339799 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.340638 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.340842 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.340898 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.343778 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:56981: BlockingRecv error: recv got EOF from 127.5.98.193:56981 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:23.469005 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:23.464573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:48973)
0924 15:28:23.464697 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:23.464707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:23.464757 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:23.464879 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:23.464880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:23.465042 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:23.465092 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.465093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.465732 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.465733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.466246 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.466247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.466578 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.466579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.466620 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.467128 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:23.467133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:23.468948 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260924 15:28:23.469151 14707 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:23.464638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:48973 (local address 127.5.98.254:41601)
0924 15:28:23.464719 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:23.464721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:23.464745 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:28:23.464819 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:23.464819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:23.464831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:23.464867 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:23.465151 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.465662 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.466314 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.466512 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.466563 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.469063 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:48973: BlockingRecv error: recv got EOF from 127.5.98.194:48973 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:23 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:23.620352 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:23.615952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:39891)
0924 15:28:23.616056 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:23.616067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:23.616129 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:23.616255 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:23.616255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:23.616409 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:23.616458 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.616460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.617070 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.617071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.617581 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:23.617582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.617887 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.617888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:23.617927 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.618419 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:23.618426 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:23.620254 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
W20260924 15:28:23.620402 14707 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:23.615997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:39891 (local address 127.5.98.254:41601)
0924 15:28:23.616063 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:23.616065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:23.616125 (+ 60us) server_negotiation.cc:407] Connection header received
0924 15:28:23.616180 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:23.616181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:23.616194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:23.616248 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:23.616512 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.617002 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.617646 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:23.617824 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:23.617873 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:23.620335 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:39891: BlockingRecv error: recv got EOF from 127.5.98.195:39891 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:24.348961 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:24.344242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:36779)
0924 15:28:24.344407 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:24.344422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:24.344498 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:24.344701 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:24.344703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:24.344932 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:24.345014 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.345017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.345686 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.345687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.346210 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.346211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.346568 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.346569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.346609 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.347039 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:24.347044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:24.348893 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
W20260924 15:28:24.349138 14711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:24.344295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:36779 (local address 127.5.98.254:41601)
0924 15:28:24.344432 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:24.344434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:24.344503 (+ 69us) server_negotiation.cc:407] Connection header received
0924 15:28:24.344550 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:24.344551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:24.344565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:24.344659 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:24.345092 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.345612 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.346293 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.346493 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.346555 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.349042 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:36779: BlockingRecv error: recv got EOF from 127.5.98.193:36779 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:24.474121 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:24.469604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:33085)
0924 15:28:24.469755 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:24.469766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:24.469816 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:24.469949 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:24.469950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:24.470113 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:24.470161 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.470163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.470782 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.470782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.471292 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.471293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.471604 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.471604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.471644 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.472107 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:24.472111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:24.474023 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
W20260924 15:28:24.474177 14711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:24.469737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:33085 (local address 127.5.98.254:41601)
0924 15:28:24.469809 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:24.469811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:24.469821 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:24.469870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:24.469870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:24.469881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:24.469928 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:24.470227 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.470716 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.471360 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.471542 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.471593 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.474115 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:33085: BlockingRecv error: recv got EOF from 127.5.98.194:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:24 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:24.625286 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:24.620963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:58153)
0924 15:28:24.621138 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:24.621149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:24.621204 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:24.621341 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:24.621342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:24.621503 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:24.621573 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.621575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.622190 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.622190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.622720 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:24.622721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.623030 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.623031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:24.623071 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.623484 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:24.623489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:24.625193 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
W20260924 15:28:24.625340 14711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:24.621045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:58153 (local address 127.5.98.254:41601)
0924 15:28:24.621115 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:24.621117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:24.621206 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:28:24.621262 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:24.621263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:24.621276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:24.621318 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:24.621634 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.622119 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.622790 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:24.622969 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:24.623016 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:24.625274 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:58153: BlockingRecv error: recv got EOF from 127.5.98.195:58153 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:25.353760 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:25.349391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:35751)
0924 15:28:25.349529 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:25.349541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:25.349591 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:25.349741 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:25.349742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:25.349900 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:25.349947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.349949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.350592 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.350593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.351129 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.351131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.351444 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.351444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.351482 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.351937 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:25.351941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:25.353663 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260924 15:28:25.353816 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:25.349458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:35751 (local address 127.5.98.254:41601)
0924 15:28:25.349590 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:25.349592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:25.349600 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:28:25.349652 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:25.349652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:25.349663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:25.349723 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:25.350008 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.350519 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.351199 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.351383 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.351431 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.353743 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:35751: BlockingRecv error: recv got EOF from 127.5.98.193:35751 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:25.479370 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:25.474598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:57743)
0924 15:28:25.474787 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:25.474798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:25.474880 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:25.475040 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:25.475041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:25.475238 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:25.475286 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.475288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.476032 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.476034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.476562 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.476564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.476950 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.476951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.477014 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.477520 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:25.477525 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:25.479264 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":104,"threads_started":1}
W20260924 15:28:25.479409 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:25.474682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:57743 (local address 127.5.98.254:41601)
0924 15:28:25.474754 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:25.474756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:25.474881 (+ 125us) server_negotiation.cc:407] Connection header received
0924 15:28:25.474961 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:25.474962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:25.474975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:25.475022 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:25.475370 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.475948 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.476659 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.476862 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.476950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.479345 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:57743: BlockingRecv error: recv got EOF from 127.5.98.194:57743 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:25 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.195@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:25.630841 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:25.625942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.195:42143)
0924 15:28:25.626240 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:25.626253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:25.626347 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:25.626521 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:25.626522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:25.626710 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:25.626757 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.626758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.627424 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.627425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.627995 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:25.627997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.628384 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.628385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:25.628445 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.628968 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:25.628972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:25.630732 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
W20260924 15:28:25.630875 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:25.626045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:42143 (local address 127.5.98.254:41601)
0924 15:28:25.626227 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:25.626229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:25.626327 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:28:25.626423 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:25.626424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:25.626440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:25.626515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:25.626837 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.627339 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.628085 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:25.628299 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:25.628373 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:25.630810 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.195:42143: BlockingRecv error: recv got EOF from 127.5.98.195:42143 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 24 15:28:26 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:26 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.193@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:26.358959 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.354253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.193:48927)
0924 15:28:26.354385 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:26.354395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:26.354439 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:26.354621 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:26.354622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:26.354771 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:26.354820 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.354822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.355466 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.355466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.356033 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.356036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.356379 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.356380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.356420 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.356981 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:26.356987 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:26.358894 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20260924 15:28:26.359040 14719 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:26.354364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:48927 (local address 127.5.98.254:41601)
0924 15:28:26.354499 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:26.354501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:26.354508 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:26.354530 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:26.354530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:26.354540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:26.354605 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:26.354885 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.355400 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.356111 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.356311 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.356365 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.358986 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.193:48927: BlockingRecv error: recv got EOF from 127.5.98.193:48927 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
Sep 24 15:28:26 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:28:26 dist-test-slave-4g6f krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.98.194@KRBTEST.COM for oryx/127.5.98.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:28:26.484627 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.479919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:41601 (local address 127.5.98.194:35797)
0924 15:28:26.480102 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:26.480115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:26.480187 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:26.480352 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:26.480353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:26.480546 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:26.480595 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.480597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.481283 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.481284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.481840 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.481842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.482232 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.482233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.482294 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.482740 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:26.482745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:26.484518 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.98.254:41601: Server oryx/127.5.98.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260924 15:28:26.484679 14719 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:26.480023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:35797 (local address 127.5.98.254:41601)
0924 15:28:26.480098 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:26.480101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:26.480167 (+ 66us) server_negotiation.cc:407] Connection header received
0924 15:28:26.480247 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:26.480248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:26.480264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:26.480329 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:26.480679 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.481191 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.481946 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.482146 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.482232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.484602 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.98.194:35797: BlockingRecv error: recv got EOF from 127.5.98.194:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:28:26.539445 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14244
I20260924 15:28:26.544327 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14396
I20260924 15:28:26.548525 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14528
I20260924 15:28:26.552601 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14174
2026-09-24T15:28:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17477 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T15:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:26Z Disabled control of system clock
I20260924 15:28:26.581213 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36311
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:28:26.662972 14724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:26.663178 14724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:26.663200 14724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:26.664484 14724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:26.664526 14724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:26.664541 14724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:26.664556 14724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:26.664568 14724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:26.665947 14724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:26.666133 14724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:26.666390 14724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:26.671518 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:26.671674 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:26.671720 14724 server_base.cc:1034] running on GCE node
W20260924 15:28:26.671622 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:26.672081 14724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:26.672346 14724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:26.673516 14724 hybrid_clock.cc:648] HybridClock initialized: now 1790263706673491 us; error 45 us; skew 500 ppm
I20260924 15:28:26.674804 14724 webserver.cc:460] Webserver started at http://127.5.98.254:44107/ using document root <none> and password file <none>
I20260924 15:28:26.675060 14724 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:26.675124 14724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:26.675228 14724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:26.676363 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "3b609b3f34184dec9bec7ab560880419"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "8f837c01666e329684fc77f3ee86fd1a"
server_key_iv: "2b1d6f83a3c6a786503995254f73588a"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.676698 14724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "3b609b3f34184dec9bec7ab560880419"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "8f837c01666e329684fc77f3ee86fd1a"
server_key_iv: "2b1d6f83a3c6a786503995254f73588a"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.678063 14724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:28:26.678789 14738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.678946 14724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:26.679028 14724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "3b609b3f34184dec9bec7ab560880419"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "8f837c01666e329684fc77f3ee86fd1a"
server_key_iv: "2b1d6f83a3c6a786503995254f73588a"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.679086 14724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:26.689682 14724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:26.689927 14724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:26.690035 14724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:26.694223 14724 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36311
I20260924 15:28:26.694250 14790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36311 every 8 connection(s)
I20260924 15:28:26.694602 14724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:26.695217 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14724
I20260924 15:28:26.695256 14791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:26.695302 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:26.695541 5515 external_mini_cluster.cc:1428] Setting key a5a9562b4c4418bcaed65dd9c4acd730
I20260924 15:28:26.699175 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419: Bootstrap starting.
I20260924 15:28:26.699839 14791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:26.700171 14791 log.cc:824] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:26.700946 14791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419: No bootstrap required, opened a new log
I20260924 15:28:26.701932 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.696117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.5.98.254:36311)
0924 15:28:26.696347 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:26.696357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:26.696390 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:28:26.696955 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:26.696964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:26.697050 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:26.697200 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:26.697451 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.698797 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.699669 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.700293 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.700424 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.701213 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:26.701217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:26.701217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:26.701411 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:26.701453 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:26.701468 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:26.701541 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:26.701634 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:28:26.701714 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260924 15:28:26.703269 14791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } }
I20260924 15:28:26.703397 14791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:26.703418 14791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b609b3f34184dec9bec7ab560880419, State: Initialized, Role: FOLLOWER
I20260924 15:28:26.703557 14791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } }
I20260924 15:28:26.703646 14791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:26.703737 14791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:26.703787 14791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:26.704299 14791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } }
I20260924 15:28:26.704413 14791 leader_election.cc:302] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b609b3f34184dec9bec7ab560880419; no voters:
I20260924 15:28:26.704586 14791 leader_election.cc:288] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:26.704625 14796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:26.704844 14791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:26.704842 14796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [term 1 LEADER]: Becoming Leader. State: Replica: 3b609b3f34184dec9bec7ab560880419, State: Running, Role: LEADER
I20260924 15:28:26.705070 14796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } }
I20260924 15:28:26.705480 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b609b3f34184dec9bec7ab560880419. Latest consensus state: current_term: 1 leader_uuid: "3b609b3f34184dec9bec7ab560880419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } } }
I20260924 15:28:26.705524 14797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b609b3f34184dec9bec7ab560880419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b609b3f34184dec9bec7ab560880419" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36311 } } }
I20260924 15:28:26.705585 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:26.705592 14797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:26.705929 14803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:26.706439 14803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:26.708057 14803 catalog_manager.cc:1347] Generated new cluster ID: 7d6562e1996a4f3083767e0c895b0a7d
I20260924 15:28:26.708112 14803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:28:26.716663 14803 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:26.717131 14803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:26.723531 14803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b609b3f34184dec9bec7ab560880419: Generated new TSK 0
I20260924 15:28:26.723732 14803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:28:26.725682 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--builtin_ntp_servers=127.5.98.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 15:28:26.807204 14815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:26.807396 14815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:26.807417 14815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:26.808583 14815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:26.808622 14815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:26.808647 14815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:26.809945 14815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:26.810176 14815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:26.810487 14815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:26.815408 14820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:26.815621 14823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:26.815624 14821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:26.816076 14815 server_base.cc:1034] running on GCE node
I20260924 15:28:26.816275 14815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:26.816517 14815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:26.817682 14815 hybrid_clock.cc:648] HybridClock initialized: now 1790263706817667 us; error 35 us; skew 500 ppm
I20260924 15:28:26.819092 14815 webserver.cc:460] Webserver started at http://127.5.98.193:34399/ using document root <none> and password file <none>
I20260924 15:28:26.819334 14815 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:26.819388 14815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:26.819520 14815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:26.820983 14815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "97b66b2c3b9b4167a90a39e17eeb513b"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "048ab0ddb0fcc11da9cecc4ee63a6f6d"
server_key_iv: "2d6d9a6775a5febd6f2c6c11dd525cef"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.821415 14815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "97b66b2c3b9b4167a90a39e17eeb513b"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "048ab0ddb0fcc11da9cecc4ee63a6f6d"
server_key_iv: "2d6d9a6775a5febd6f2c6c11dd525cef"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.823007 14815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:28:26.824070 14829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.824295 14815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:26.824386 14815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "97b66b2c3b9b4167a90a39e17eeb513b"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "048ab0ddb0fcc11da9cecc4ee63a6f6d"
server_key_iv: "2d6d9a6775a5febd6f2c6c11dd525cef"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.824455 14815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:26.860818 14815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:26.861088 14815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:26.861192 14815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:26.861374 14815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:26.861637 14815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:26.861667 14815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.861688 14815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:26.861701 14815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.867522 14815 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:45139
I20260924 15:28:26.867585 14942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:45139 every 8 connection(s)
I20260924 15:28:26.867969 14815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:26.870966 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14815
I20260924 15:28:26.871064 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:26.871259 5515 external_mini_cluster.cc:1428] Setting key 2ea09af79ad6eb3783e4e664cc104547
I20260924 15:28:26.872274 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--builtin_ntp_servers=127.5.98.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 15:28:26.873703 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.868726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:46335 (local address 127.5.98.254:36311)
0924 15:28:26.868815 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:26.868817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:26.869022 (+ 205us) server_negotiation.cc:407] Connection header received
0924 15:28:26.869530 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:26.869531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:26.869544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:26.869594 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:26.870230 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.870726 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.872157 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.872339 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.872424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.873384 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:26.873385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:26.873386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:26.873391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:26.873411 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:26.873416 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:26.873480 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:26.873583 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 15:28:26.873624 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:28:26.873942 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.868695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36311 (local address 127.5.98.193:46335)
0924 15:28:26.868890 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:26.868923 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:26.869415 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:26.869658 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:26.869661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:26.869669 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:26.870147 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.870151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.870823 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.870824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.872069 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.872072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.872492 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.872493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.872658 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.873225 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:26.873226 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:26.873261 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:26.873287 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:26.873497 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:26.873504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:26.873508 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:26.873575 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:28:26.873691 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260924 15:28:26.874387 14943 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36311
I20260924 15:28:26.874500 14943 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:26.874680 14943 heartbeater.cc:511] Master 127.5.98.254:36311 requested a full tablet report, sending...
I20260924 15:28:26.875240 14755 ts_manager.cc:194] Registered new tserver with Master: 97b66b2c3b9b4167a90a39e17eeb513b (127.5.98.193:45139)
I20260924 15:28:26.875881 14755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.193:46335
W20260924 15:28:26.953048 14946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:26.953269 14946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:26.953302 14946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:26.954401 14946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:26.954469 14946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:26.954512 14946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:26.955933 14946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:26.956174 14946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:26.956456 14946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:26.961405 14951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:26.961385 14952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:26.961351 14954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:26.961619 14946 server_base.cc:1034] running on GCE node
I20260924 15:28:26.961822 14946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:26.962015 14946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:26.963150 14946 hybrid_clock.cc:648] HybridClock initialized: now 1790263706963118 us; error 48 us; skew 500 ppm
I20260924 15:28:26.964303 14946 webserver.cc:460] Webserver started at http://127.5.98.194:46143/ using document root <none> and password file <none>
I20260924 15:28:26.964540 14946 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:26.964587 14946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:26.964744 14946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:26.965794 14946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "e62a061656854592926b2166c90d2f01"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "89456d44af287634741b8934d6cb5bab"
server_key_iv: "37044106e7bfc40b723adf71c2b7f409"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.966152 14946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "e62a061656854592926b2166c90d2f01"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "89456d44af287634741b8934d6cb5bab"
server_key_iv: "37044106e7bfc40b723adf71c2b7f409"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.967446 14946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:26.968245 14960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.968425 14946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.968521 14946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "e62a061656854592926b2166c90d2f01"
format_stamp: "Formatted at 2026-09-24 15:28:26 on dist-test-slave-4g6f"
server_key: "89456d44af287634741b8934d6cb5bab"
server_key_iv: "37044106e7bfc40b723adf71c2b7f409"
server_key_version: "encryptionkey@0"
I20260924 15:28:26.968611 14946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:26.984038 14946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:26.984330 14946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:26.984481 14946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:26.984695 14946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:26.985013 14946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:26.985065 14946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.985112 14946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:26.985140 14946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:26.990702 14946 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40465
I20260924 15:28:26.990787 15073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40465 every 8 connection(s)
I20260924 15:28:26.991119 14946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:28:26.996912 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.991919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:44815 (local address 127.5.98.254:36311)
0924 15:28:26.991967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:26.991970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:26.992199 (+ 229us) server_negotiation.cc:407] Connection header received
0924 15:28:26.992816 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:26.992818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:26.992831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:26.992892 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:26.993632 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.994187 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.995286 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.995538 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.995635 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.996600 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:26.996601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:26.996602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:26.996608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:26.996625 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:26.996629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:26.996688 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:26.996796 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:28:26.996827 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:28:26.997156 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:26.991883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36311 (local address 127.5.98.194:44815)
0924 15:28:26.992071 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:26.992106 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:26.992660 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:26.992980 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:26.992987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:26.993002 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:26.993533 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.993537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.994298 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.994299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.995171 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:26.995174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:26.995647 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:26.995648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:26.995824 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:26.996408 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:26.996409 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:26.996439 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:26.996463 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:26.996695 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:26.996700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:26.996703 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:26.996800 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 15:28:26.996909 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 15:28:26.997716 15074 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36311
I20260924 15:28:26.997742 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 14946
I20260924 15:28:26.997800 15074 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:26.997869 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:26.998023 15074 heartbeater.cc:511] Master 127.5.98.254:36311 requested a full tablet report, sending...
I20260924 15:28:26.998077 5515 external_mini_cluster.cc:1428] Setting key a36f476e85025c1e5e31a31efce17181
I20260924 15:28:26.998441 14755 ts_manager.cc:194] Registered new tserver with Master: e62a061656854592926b2166c90d2f01 (127.5.98.194:40465)
I20260924 15:28:26.998903 14755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.194:44815
I20260924 15:28:26.999183 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--builtin_ntp_servers=127.5.98.212:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 15:28:27.081916 15077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:27.082173 15077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:27.082211 15077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:27.083796 15077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:27.083853 15077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:27.083892 15077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:27.085201 15077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:27.085467 15077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:27.085777 15077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:27.090746 15082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.090736 15083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.090749 15085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.091029 15077 server_base.cc:1034] running on GCE node
I20260924 15:28:27.091236 15077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:27.091470 15077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:27.092636 15077 hybrid_clock.cc:648] HybridClock initialized: now 1790263707092619 us; error 49 us; skew 500 ppm
I20260924 15:28:27.093945 15077 webserver.cc:460] Webserver started at http://127.5.98.195:36023/ using document root <none> and password file <none>
I20260924 15:28:27.094197 15077 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:27.094265 15077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:27.094404 15077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:27.095465 15077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "642b1881397d4e749ad39dbc774f5cbc"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "3514c7c8dac491c612708cb9867c42cf"
server_key_iv: "3e5defcedbcdbe042415c14e62a42380"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.095876 15077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "642b1881397d4e749ad39dbc774f5cbc"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "3514c7c8dac491c612708cb9867c42cf"
server_key_iv: "3e5defcedbcdbe042415c14e62a42380"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.097594 15077 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 15:28:27.098285 15091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.098568 15077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:27.098665 15077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "642b1881397d4e749ad39dbc774f5cbc"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "3514c7c8dac491c612708cb9867c42cf"
server_key_iv: "3e5defcedbcdbe042415c14e62a42380"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.098747 15077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:27.108111 15077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:27.108364 15077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:27.108498 15077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:27.108696 15077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:27.108994 15077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:27.109053 15077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.109092 15077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:27.109118 15077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.114683 15077 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:38529
I20260924 15:28:27.114724 15204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:38529 every 8 connection(s)
I20260924 15:28:27.115022 15077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:27.119685 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.115685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:36459 (local address 127.5.98.254:36311)
0924 15:28:27.115726 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.115729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.115965 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:28:27.116401 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.116402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.116414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.116466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.117086 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:27.117561 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:27.118483 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:27.118706 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:27.118760 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:27.119445 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:27.119446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:27.119447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:27.119451 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:27.119466 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:27.119469 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:27.119512 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:27.119571 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:28:27.119593 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:28:27.119923 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.115639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36311 (local address 127.5.98.195:36459)
0924 15:28:27.115842 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:27.115872 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:27.116317 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:27.116516 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:27.116519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:27.116527 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:27.117012 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:27.117016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:27.117635 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:27.117636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:27.118398 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:27.118400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:27.118777 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:27.118778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:27.118882 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:27.119316 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:27.119317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:27.119344 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:27.119365 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:27.119512 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:27.119514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:27.119516 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:28:27.119562 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:28:27.119634 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":122,"threads_started":1}
I20260924 15:28:27.120432 15205 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36311
I20260924 15:28:27.120532 15205 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:27.120726 15205 heartbeater.cc:511] Master 127.5.98.254:36311 requested a full tablet report, sending...
I20260924 15:28:27.121083 14755 ts_manager.cc:194] Registered new tserver with Master: 642b1881397d4e749ad39dbc774f5cbc (127.5.98.195:38529)
I20260924 15:28:27.121435 14755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.195:36459
I20260924 15:28:27.123888 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15077
I20260924 15:28:27.123965 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:27.124116 5515 external_mini_cluster.cc:1428] Setting key 1f3eede2f0eebbec385aa693ac5668e5
I20260924 15:28:27.124956 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:28:27.126755 14794 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:27.126223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.5.98.254:36311)
0924 15:28:27.126290 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.126292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.126411 (+ 119us) server_negotiation.cc:407] Connection header received
0924 15:28:27.126478 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.126479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.126488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.126556 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.126698 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47588: BlockingRecv error: recv got EOF from 127.0.0.1:47588 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:28:27.126827 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14815
I20260924 15:28:27.133054 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14946
I20260924 15:28:27.137735 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15077
I20260924 15:28:27.141871 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 14724
2026-09-24T15:28:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (586 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T15:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:27Z Disabled control of system clock
I20260924 15:28:27.167379 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40129
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:40129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:28:27.246063 15216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:27.246249 15216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:27.246269 15216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:27.247292 15216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:27.247324 15216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:27.247336 15216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:27.247349 15216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:27.247361 15216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:27.248622 15216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:40129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:27.248795 15216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:27.249012 15216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:27.253618 15222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.253638 15221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.253769 15216 server_base.cc:1034] running on GCE node
W20260924 15:28:27.253643 15224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.254068 15216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:27.254338 15216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:27.255499 15216 hybrid_clock.cc:648] HybridClock initialized: now 1790263707255497 us; error 66 us; skew 500 ppm
I20260924 15:28:27.256709 15216 webserver.cc:460] Webserver started at http://127.5.98.254:44179/ using document root <none> and password file <none>
I20260924 15:28:27.256949 15216 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:27.257017 15216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:27.257162 15216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:27.258158 15216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "66a532ac06b74f9da33abb081f9c1df4"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c98cfd73cd9831416db428f9a4c408b1"
server_key_iv: "9866b07e4fe0c32c3c7a77df4586a5ad"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.258522 15216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "66a532ac06b74f9da33abb081f9c1df4"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c98cfd73cd9831416db428f9a4c408b1"
server_key_iv: "9866b07e4fe0c32c3c7a77df4586a5ad"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.259835 15216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:27.260535 15230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.260699 15216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.260803 15216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "66a532ac06b74f9da33abb081f9c1df4"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c98cfd73cd9831416db428f9a4c408b1"
server_key_iv: "9866b07e4fe0c32c3c7a77df4586a5ad"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.260882 15216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:27.264119 15216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:27.264371 15216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:27.264523 15216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:27.268136 15216 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:40129
I20260924 15:28:27.268167 15282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:40129 every 8 connection(s)
I20260924 15:28:27.268481 15216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:27.269142 15283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:27.271059 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15216
I20260924 15:28:27.271164 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:27.271337 5515 external_mini_cluster.cc:1428] Setting key e3a6d759e7b21b6b479e02d38eee229b
I20260924 15:28:27.272214 15283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4: Bootstrap starting.
I20260924 15:28:27.272886 15283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:27.273219 15283 log.cc:824] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:27.273983 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.271875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.5.98.254:40129)
0924 15:28:27.272072 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.272081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.272111 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:28:27.272664 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.272673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.272723 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.272892 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.273494 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:27.273497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:27.273497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:27.273502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:27.273552 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:27.273563 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:27.273622 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:27.273739 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:28:27.273794 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 15:28:27.274046 15283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4: No bootstrap required, opened a new log
I20260924 15:28:27.275457 15283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } }
I20260924 15:28:27.275583 15283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:27.275604 15283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a532ac06b74f9da33abb081f9c1df4, State: Initialized, Role: FOLLOWER
I20260924 15:28:27.275758 15283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } }
I20260924 15:28:27.275847 15283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:27.275893 15283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:27.275940 15283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:27.276680 15283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } }
I20260924 15:28:27.276793 15283 leader_election.cc:302] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a532ac06b74f9da33abb081f9c1df4; no voters:
I20260924 15:28:27.276947 15283 leader_election.cc:288] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:27.277029 15288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:27.277174 15288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [term 1 LEADER]: Becoming Leader. State: Replica: 66a532ac06b74f9da33abb081f9c1df4, State: Running, Role: LEADER
I20260924 15:28:27.277207 15283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:27.277269 15288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } }
I20260924 15:28:27.277686 15289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a532ac06b74f9da33abb081f9c1df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } } }
I20260924 15:28:27.277814 15289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:27.277685 15290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a532ac06b74f9da33abb081f9c1df4. Latest consensus state: current_term: 1 leader_uuid: "66a532ac06b74f9da33abb081f9c1df4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a532ac06b74f9da33abb081f9c1df4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40129 } } }
I20260924 15:28:27.277956 15290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:27.278062 15297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:27.278502 15297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:27.280047 15297 catalog_manager.cc:1347] Generated new cluster ID: 0fa5e1de96044a398a8b9303391e7c28
I20260924 15:28:27.280102 15297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:28:27.297178 15297 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:27.297315 15297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:27.300680 15297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a532ac06b74f9da33abb081f9c1df4: Generated new TSK 0
I20260924 15:28:27.300832 15297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:28:27.306519 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--builtin_ntp_servers=127.5.98.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:28:27.387265 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:27.387460 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:27.387599 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:27.388895 15307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:27.388943 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:27.388970 15307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:27.390311 15307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:27.390570 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:27.390914 15307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:27.396011 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.395984 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.396116 15307 server_base.cc:1034] running on GCE node
W20260924 15:28:27.395980 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.396489 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:27.396736 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:27.397907 15307 hybrid_clock.cc:648] HybridClock initialized: now 1790263707397890 us; error 35 us; skew 500 ppm
I20260924 15:28:27.399288 15307 webserver.cc:460] Webserver started at http://127.5.98.193:44391/ using document root <none> and password file <none>
I20260924 15:28:27.399557 15307 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:27.399616 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:27.399760 15307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:27.401078 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "d972794df1df4179b78745634afef853"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "07fa61be7088d640a3f508d24ed87c40"
server_key_iv: "41b20959273d9001c8559e5be5790f29"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.401501 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "d972794df1df4179b78745634afef853"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "07fa61be7088d640a3f508d24ed87c40"
server_key_iv: "41b20959273d9001c8559e5be5790f29"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.402767 15307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:28:27.403394 15321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.403578 15307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:27.403694 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "d972794df1df4179b78745634afef853"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "07fa61be7088d640a3f508d24ed87c40"
server_key_iv: "41b20959273d9001c8559e5be5790f29"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.403779 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:27.406985 15307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:27.407244 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:27.407387 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:27.407588 15307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:27.407912 15307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:27.407968 15307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.408007 15307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:27.408036 15307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.413533 15307 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:40313
I20260924 15:28:27.413605 15434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:40313 every 8 connection(s)
I20260924 15:28:27.413911 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:28:27.416950 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.414624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:47555 (local address 127.5.98.254:40129)
0924 15:28:27.414687 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.414689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.414893 (+ 204us) server_negotiation.cc:407] Connection header received
0924 15:28:27.415387 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.415388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.415394 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.415452 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.416702 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:27.416704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:27.416704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:27.416709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:27.416724 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:27.416728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:27.416782 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:27.416866 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:28:27.416881 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:28:27.417137 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.414606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40129 (local address 127.5.98.193:47555)
0924 15:28:27.414785 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:27.414816 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:27.415282 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:27.415533 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:27.415537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:27.415551 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:27.416544 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:27.416545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:27.416572 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:27.416600 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:27.416790 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:27.416795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:27.416798 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:27.416876 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:28:27.416935 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260924 15:28:27.417640 15435 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40129
I20260924 15:28:27.417711 15435 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:27.417860 15435 heartbeater.cc:511] Master 127.5.98.254:40129 requested a full tablet report, sending...
I20260924 15:28:27.418392 15247 ts_manager.cc:194] Registered new tserver with Master: d972794df1df4179b78745634afef853 (127.5.98.193:40313)
I20260924 15:28:27.421285 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15307
I20260924 15:28:27.421393 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:27.421587 5515 external_mini_cluster.cc:1428] Setting key 2dd04b945aa2fc6a89df22f864f2566a
I20260924 15:28:27.422407 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--builtin_ntp_servers=127.5.98.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:28:27.502104 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:27.502328 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:27.502362 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:27.503486 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:27.503548 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:27.503589 15438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:27.504984 15438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:27.505237 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:27.505602 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:27.510560 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.510607 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.510807 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.510783 15438 server_base.cc:1034] running on GCE node
I20260924 15:28:27.511008 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:27.511195 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:27.512351 15438 hybrid_clock.cc:648] HybridClock initialized: now 1790263707512337 us; error 27 us; skew 500 ppm
I20260924 15:28:27.513527 15438 webserver.cc:460] Webserver started at http://127.5.98.194:33139/ using document root <none> and password file <none>
I20260924 15:28:27.513761 15438 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:27.513826 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:27.513971 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:27.515003 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "a0ec50f5437b424ca6d4ca5a80fafe19"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c9c2240cef96b500d08e25eae6de15fd"
server_key_iv: "08d3c82fb859553bf4ca33945ee6a715"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.515373 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "a0ec50f5437b424ca6d4ca5a80fafe19"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c9c2240cef96b500d08e25eae6de15fd"
server_key_iv: "08d3c82fb859553bf4ca33945ee6a715"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.516693 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:28:27.517383 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.517555 15438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:27.517653 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "a0ec50f5437b424ca6d4ca5a80fafe19"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "c9c2240cef96b500d08e25eae6de15fd"
server_key_iv: "08d3c82fb859553bf4ca33945ee6a715"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.517736 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:27.520995 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:27.521248 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:27.521379 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:27.521582 15438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:27.521878 15438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:27.521927 15438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.521966 15438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:27.521996 15438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.527611 15438 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46251
I20260924 15:28:27.527650 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46251 every 8 connection(s)
I20260924 15:28:27.527988 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:28:27.530689 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.528678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:40835 (local address 127.5.98.254:40129)
0924 15:28:27.528730 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.528733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.528947 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:28:27.529375 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.529376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.529380 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.529432 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.530465 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:27.530467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:27.530467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:27.530471 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:27.530485 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:27.530489 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:27.530554 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:27.530624 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:28:27.530636 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:28:27.530882 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.528671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40129 (local address 127.5.98.194:40835)
0924 15:28:27.528834 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:27.528866 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:27.529296 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:27.529481 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:27.529484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:27.529492 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:27.530322 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:27.530322 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:27.530350 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:27.530379 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:27.530549 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:27.530551 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:27.530554 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:27.530631 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:28:27.530679 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260924 15:28:27.531358 15566 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40129
I20260924 15:28:27.531445 15566 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:27.531555 15566 heartbeater.cc:511] Master 127.5.98.254:40129 requested a full tablet report, sending...
I20260924 15:28:27.531908 15247 ts_manager.cc:194] Registered new tserver with Master: a0ec50f5437b424ca6d4ca5a80fafe19 (127.5.98.194:46251)
I20260924 15:28:27.536777 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15438
I20260924 15:28:27.536876 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:27.537062 5515 external_mini_cluster.cc:1428] Setting key e3e80e26c5bc9f2afaa40fc0ccf43fd7
I20260924 15:28:27.537919 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--builtin_ntp_servers=127.5.98.212:41693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:28:27.619112 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:27.619314 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:27.619338 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:27.620520 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:27.620565 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:27.620594 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:27.621923 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:27.622184 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:27.622442 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:27.627529 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.627529 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:27.627606 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:27.627705 15569 server_base.cc:1034] running on GCE node
I20260924 15:28:27.627957 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:27.628146 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:27.629281 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790263707629260 us; error 38 us; skew 500 ppm
I20260924 15:28:27.635576 15569 webserver.cc:460] Webserver started at http://127.5.98.195:45055/ using document root <none> and password file <none>
I20260924 15:28:27.635831 15569 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:27.635880 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:27.636044 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:27.637073 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "5dc7aaa7219349a8bc651b072cbb604a"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "9edb0932d5aeff2dcd00211b22fa0386"
server_key_iv: "953ae706758114fb9862b0f817c9ad1a"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.637437 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "5dc7aaa7219349a8bc651b072cbb604a"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "9edb0932d5aeff2dcd00211b22fa0386"
server_key_iv: "953ae706758114fb9862b0f817c9ad1a"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.638716 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:28:27.639410 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.639618 15569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:27.639735 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "5dc7aaa7219349a8bc651b072cbb604a"
format_stamp: "Formatted at 2026-09-24 15:28:27 on dist-test-slave-4g6f"
server_key: "9edb0932d5aeff2dcd00211b22fa0386"
server_key_iv: "953ae706758114fb9862b0f817c9ad1a"
server_key_version: "encryptionkey@0"
I20260924 15:28:27.639812 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:27.643069 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:27.643321 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:27.643455 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:27.643716 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:27.644037 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:27.644069 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.644120 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:27.644147 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:27.649513 15569 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42963
I20260924 15:28:27.649612 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42963 every 8 connection(s)
I20260924 15:28:27.649901 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:27.652676 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15569
I20260924 15:28:27.652726 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.650608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:40423 (local address 127.5.98.254:40129)
0924 15:28:27.650661 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.650664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.650904 (+ 240us) server_negotiation.cc:407] Connection header received
0924 15:28:27.651329 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.651329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.651335 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.651386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.652472 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:27.652473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:27.652473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:28:27.652477 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:27.652491 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:28:27.652495 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:27.652549 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:27.652662 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 15:28:27.652675 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:28:27.652778 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:27.652956 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:27.650604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40129 (local address 127.5.98.195:40423)
0924 15:28:27.650786 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:27.650818 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:27.651248 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:27.651437 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:27.651440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:27.651447 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:27.652316 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:28:27.652317 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:27.652344 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:28:27.652371 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:27.652569 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:27.652575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:27.652578 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:28:27.652669 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:28:27.652735 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 15:28:27.652984 5515 external_mini_cluster.cc:1428] Setting key b4f12318ff84d507e72a0b3108d029ac
I20260924 15:28:27.653476 15697 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40129
I20260924 15:28:27.653570 15697 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:27.653717 15697 heartbeater.cc:511] Master 127.5.98.254:40129 requested a full tablet report, sending...
I20260924 15:28:27.654064 15247 ts_manager.cc:194] Registered new tserver with Master: 5dc7aaa7219349a8bc651b072cbb604a (127.5.98.195:42963)
I20260924 15:28:27.664805 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:28:27.666639 15286 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:28:27.666126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.5.98.254:40129)
0924 15:28:27.666194 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:27.666196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:27.666344 (+ 148us) server_negotiation.cc:407] Connection header received
0924 15:28:27.666385 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:27.666386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:27.666390 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:27.666435 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:27.666565 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55290: BlockingRecv error: recv got EOF from 127.0.0.1:55290 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:28:27.666695 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15307
I20260924 15:28:27.671726 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15438
I20260924 15:28:27.675961 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15569
I20260924 15:28:27.680456 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15216
2026-09-24T15:28:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (538 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:27 dist-test-slave-4g6f krb5kdc[15707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:27 dist-test-slave-4g6f krb5kdc[15707](info): set up 2 sockets
Sep 24 15:28:27 dist-test-slave-4g6f krb5kdc[15707](info): commencing operation
krb5kdc: starting...
W20260924 15:28:28.781543 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:28 dist-test-slave-4g6f krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:28:28.897828 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40761
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:28.977108 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:28.977303 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:28.977324 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:28.978381 15720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:28:28.978416 15720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:28.978430 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:28.978443 15720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:28:28.978456 15720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:28:28.979858 15720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:40761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:28.980036 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:28.980201 15720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:28.984875 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:28.984912 15725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:28.984918 15726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:28.985006 15720 server_base.cc:1034] running on GCE node
I20260924 15:28:28.985368 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:28.985756 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:28.986949 15720 hybrid_clock.cc:648] HybridClock initialized: now 1790263708986932 us; error 32 us; skew 500 ppm
Sep 24 15:28:28 dist-test-slave-4g6f krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263708, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:28.989548 15720 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:28:28.990018 15720 webserver.cc:460] Webserver started at http://127.5.98.254:42099/ using document root <none> and password file <none>
I20260924 15:28:28.990268 15720 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:28.990337 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:28.990482 15720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:28.991542 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "20264369b4ea4466b3471f70f1aec286"
format_stamp: "Formatted at 2026-09-24 15:28:28 on dist-test-slave-4g6f"
server_key: "534fce4aba6408fad7777bae0cea5d17"
server_key_iv: "ff70ffc5ea29e0155b27d2c86fa24d9e"
server_key_version: "encryptionkey@0"
I20260924 15:28:28.991981 15720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "20264369b4ea4466b3471f70f1aec286"
format_stamp: "Formatted at 2026-09-24 15:28:28 on dist-test-slave-4g6f"
server_key: "534fce4aba6408fad7777bae0cea5d17"
server_key_iv: "ff70ffc5ea29e0155b27d2c86fa24d9e"
server_key_version: "encryptionkey@0"
I20260924 15:28:28.993259 15720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:28:28.993948 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:28.994172 15720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:28:28.994282 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "20264369b4ea4466b3471f70f1aec286"
format_stamp: "Formatted at 2026-09-24 15:28:28 on dist-test-slave-4g6f"
server_key: "534fce4aba6408fad7777bae0cea5d17"
server_key_iv: "ff70ffc5ea29e0155b27d2c86fa24d9e"
server_key_version: "encryptionkey@0"
I20260924 15:28:28.994364 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:29.009730 15720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:29.011626 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:29.011826 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:29.015713 15720 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:40761
I20260924 15:28:29.015761 15787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:40761 every 8 connection(s)
I20260924 15:28:29.016147 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:28:29.017118 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:29.019321 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: Bootstrap starting.
I20260924 15:28:29.019919 15788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:29.020157 15788 log.cc:824] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:29.020803 15788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: No bootstrap required, opened a new log
I20260924 15:28:29.022320 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15720
I20260924 15:28:29.022415 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:28:29.022617 5515 external_mini_cluster.cc:1428] Setting key 7965e460904e22d0fd5d518426c0773d
I20260924 15:28:29.022521 15788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } }
I20260924 15:28:29.022688 15788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:29.022737 15788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20264369b4ea4466b3471f70f1aec286, State: Initialized, Role: FOLLOWER
I20260924 15:28:29.022876 15788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } }
I20260924 15:28:29.022970 15788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:28:29.023018 15788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:28:29.023080 15788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:29.023741 15788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } }
I20260924 15:28:29.023862 15788 leader_election.cc:302] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20264369b4ea4466b3471f70f1aec286; no voters:
I20260924 15:28:29.024050 15788 leader_election.cc:288] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:28:29.024067 15793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:29.024173 15793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [term 1 LEADER]: Becoming Leader. State: Replica: 20264369b4ea4466b3471f70f1aec286, State: Running, Role: LEADER
I20260924 15:28:29.024260 15793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } }
I20260924 15:28:29.024322 15788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:28:29.024585 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20264369b4ea4466b3471f70f1aec286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } } }
I20260924 15:28:29.024835 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:29.024762 15794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20264369b4ea4466b3471f70f1aec286. Latest consensus state: current_term: 1 leader_uuid: "20264369b4ea4466b3471f70f1aec286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20264369b4ea4466b3471f70f1aec286" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40761 } } }
I20260924 15:28:29.025098 15794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286 [sys.catalog]: This master's current role is: LEADER
I20260924 15:28:29.025260 15802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:28:29.025789 15802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:28:29.027424 15802 catalog_manager.cc:1347] Generated new cluster ID: 6062f29c245a42af8c934c72aab4c498
I20260924 15:28:29.027477 15802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:29.034403 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.023144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.5.98.254:40761)
0924 15:28:29.023313 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.023323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.023362 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:28:29.023696 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.023703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.023748 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.023899 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:29.024728 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.026162 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.027222 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.027834 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.027930 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.030817 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:29.030820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:29.030820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:29.030831 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:29.032759 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.033179 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.033182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.033182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.033292 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.033520 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.033520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.033521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.033809 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:29.033940 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.034085 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.034211 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260924 15:28:29.041110 15802 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:28:29.041628 15802 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:28:29.045303 15802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: Generated new TSK 0
I20260924 15:28:29.045477 15802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:28:29.072120 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:29.155164 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:29.155386 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:29.155426 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:29.156580 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:29.156647 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:29.156694 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:28:29.158149 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:29.158418 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:29.158653 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:29.163602 15822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:29.163637 15820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:29.163619 15819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:29.163813 15814 server_base.cc:1034] running on GCE node
I20260924 15:28:29.164076 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:29.164270 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:29.165405 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790263709165400 us; error 31 us; skew 500 ppm
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:29.167934 15814 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:28:29.168350 15814 webserver.cc:460] Webserver started at http://127.5.98.193:39953/ using document root <none> and password file <none>
I20260924 15:28:29.168596 15814 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:29.168643 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:29.168807 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:29.169812 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "567d921ece9c4877b2311a8d42600675"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b9dd4ac724408df8b317e6d1421a0215"
server_key_iv: "a28ac6fea1cc70c05c7b18d204f015b6"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.170154 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "567d921ece9c4877b2311a8d42600675"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b9dd4ac724408df8b317e6d1421a0215"
server_key_iv: "a28ac6fea1cc70c05c7b18d204f015b6"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.171415 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:28:29.172210 15829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.172379 15814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.172477 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "567d921ece9c4877b2311a8d42600675"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b9dd4ac724408df8b317e6d1421a0215"
server_key_iv: "a28ac6fea1cc70c05c7b18d204f015b6"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.172571 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:29.188391 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:29.190208 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:29.190387 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:29.190618 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:29.190933 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:29.190986 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.191021 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:29.191049 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.196787 15814 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:38075
I20260924 15:28:29.196897 15942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:38075 every 8 connection(s)
I20260924 15:28:29.197194 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:29.205962 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.197979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:36231 (local address 127.5.98.254:40761)
0924 15:28:29.198057 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.198060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.198287 (+ 227us) server_negotiation.cc:407] Connection header received
0924 15:28:29.198784 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.198785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.198799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.198852 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:29.199813 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.200303 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.201241 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.201426 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.201475 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.203420 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:29.203422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:29.203422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:29.203428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:29.204854 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.205203 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.205204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.205204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.205254 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.205459 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.205459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.205459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.205557 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:29.205614 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.205836 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.205898 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:28:29.206171 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.197943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40761 (local address 127.5.98.193:36231)
0924 15:28:29.198126 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.198162 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.198693 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.198909 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.198912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.199475 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:29.199731 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.199736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.200379 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.200379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.201169 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.201171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.201496 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.201497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.201608 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.202058 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:29.202066 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:29.203338 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:29.204920 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.204925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.204927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.205137 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.205312 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.205313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.205313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.205405 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.205619 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:29.205623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:29.205728 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.205808 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.205921 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260924 15:28:29.206777 15943 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40761
I20260924 15:28:29.206884 15943 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:29.207069 15943 heartbeater.cc:511] Master 127.5.98.254:40761 requested a full tablet report, sending...
I20260924 15:28:29.207125 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15814
I20260924 15:28:29.207252 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:28:29.207465 5515 external_mini_cluster.cc:1428] Setting key 93f760ed0e6aa7d2993dccfb6830283f
I20260924 15:28:29.207610 15752 ts_manager.cc:194] Registered new tserver with Master: 567d921ece9c4877b2311a8d42600675 (127.5.98.193:38075)
I20260924 15:28:29.208433 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:36231
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:28:29.231926 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:29.312218 15948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:29.312404 15948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:29.312426 15948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:29.313460 15948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:29.313495 15948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:29.313535 15948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:28:29.314883 15948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:29.315063 15948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:29.315243 15948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:29.320147 15954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:29.320175 15953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:29.320225 15948 server_base.cc:1034] running on GCE node
W20260924 15:28:29.320156 15956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:29.320582 15948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:29.320824 15948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:29.322003 15948 hybrid_clock.cc:648] HybridClock initialized: now 1790263709321982 us; error 34 us; skew 500 ppm
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:29.324507 15948 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:28:29.325031 15948 webserver.cc:460] Webserver started at http://127.5.98.194:39757/ using document root <none> and password file <none>
I20260924 15:28:29.325263 15948 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:29.325309 15948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:29.325397 15948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:29.326457 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "d0a5879d2dd84d4da1136c72a6ceea4a"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "8ccd97545b6c8bac3b1b3e894a35483c"
server_key_iv: "b748b63763e02c0dfa3df14f0d5a56fa"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.326764 15948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "d0a5879d2dd84d4da1136c72a6ceea4a"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "8ccd97545b6c8bac3b1b3e894a35483c"
server_key_iv: "b748b63763e02c0dfa3df14f0d5a56fa"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.328039 15948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:28:29.328706 15963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.328881 15948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:29.328986 15948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "d0a5879d2dd84d4da1136c72a6ceea4a"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "8ccd97545b6c8bac3b1b3e894a35483c"
server_key_iv: "b748b63763e02c0dfa3df14f0d5a56fa"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.329062 15948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:29.339392 15948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:29.341156 15948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:29.341312 15948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:29.341517 15948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:29.341827 15948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:29.341881 15948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.341919 15948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:29.341948 15948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.347800 15948 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:33741
I20260924 15:28:29.347847 16076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:33741 every 8 connection(s)
I20260924 15:28:29.348178 15948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:29.356446 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 15948
I20260924 15:28:29.356532 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:28:29.356746 5515 external_mini_cluster.cc:1428] Setting key a6e7bd7e7146a186113114a3601f6216
I20260924 15:28:29.357146 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.348841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:60315 (local address 127.5.98.254:40761)
0924 15:28:29.348920 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.348922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.349090 (+ 168us) server_negotiation.cc:407] Connection header received
0924 15:28:29.349570 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.349571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.349582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.349633 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:29.350526 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.351009 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.352015 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.352207 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.352260 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.354192 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:29.354193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:29.354193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:29.354201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:29.355577 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.356034 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.356035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.356036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.356087 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.356406 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.356407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.356407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.356588 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:29.356674 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.356968 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.357054 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:28:29.357380 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.348794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40761 (local address 127.5.98.194:60315)
0924 15:28:29.348945 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.348979 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.349478 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.349713 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.349717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.350271 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:29.350451 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.350456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.351093 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.351094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.351943 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.351945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.352279 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.352279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.352373 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.352791 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:29.352799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:29.354085 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:29.355702 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.355710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.355712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.355933 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.356166 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.356167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.356168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.356318 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.356667 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:29.356674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:29.356800 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.356945 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.357109 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 15:28:29.357865 16077 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40761
I20260924 15:28:29.357954 16077 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:29.358143 16077 heartbeater.cc:511] Master 127.5.98.254:40761 requested a full tablet report, sending...
I20260924 15:28:29.358537 15752 ts_manager.cc:194] Registered new tserver with Master: d0a5879d2dd84d4da1136c72a6ceea4a (127.5.98.194:33741)
I20260924 15:28:29.358896 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:60315
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:28:29.382779 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:28:29.464334 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:28:29.464534 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:28:29.464557 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:28:29.465694 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:28:29.465735 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:28:29.465791 16082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:28:29.467305 16082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:28:29.467554 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:28:29.467806 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:28:29.472748 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:28:29.472751 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:29.472839 16082 server_base.cc:1034] running on GCE node
W20260924 15:28:29.472779 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:28:29.473181 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:28:29.473428 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:28:29.474654 16082 hybrid_clock.cc:648] HybridClock initialized: now 1790263709474614 us; error 50 us; skew 500 ppm
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:28:29.477273 16082 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:28:29.477743 16082 webserver.cc:460] Webserver started at http://127.5.98.195:33397/ using document root <none> and password file <none>
I20260924 15:28:29.477993 16082 fs_manager.cc:359] Metadata directory not provided
I20260924 15:28:29.478066 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:28:29.478221 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:28:29.479266 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "17e0df97cd5b40c7a13f800de899dfe9"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b99d9e57ac143d0bc73e9a61f9203667"
server_key_iv: "254a52a2c39a13c108e0e364f94d0e6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.479717 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "17e0df97cd5b40c7a13f800de899dfe9"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b99d9e57ac143d0bc73e9a61f9203667"
server_key_iv: "254a52a2c39a13c108e0e364f94d0e6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.481112 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:28:29.481853 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.482023 16082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:28:29.482131 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "17e0df97cd5b40c7a13f800de899dfe9"
format_stamp: "Formatted at 2026-09-24 15:28:29 on dist-test-slave-4g6f"
server_key: "b99d9e57ac143d0bc73e9a61f9203667"
server_key_iv: "254a52a2c39a13c108e0e364f94d0e6c"
server_key_version: "encryptionkey@0"
I20260924 15:28:29.482220 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:28:29.513655 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:28:29.515599 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:28:29.515796 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:28:29.516073 16082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:28:29.516448 16082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:28:29.516510 16082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.516551 16082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:28:29.516585 16082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:28:29.522828 16082 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:43749
I20260924 15:28:29.522872 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:43749 every 8 connection(s)
I20260924 15:28:29.523226 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:28:29.527613 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 16082
I20260924 15:28:29.527736 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:28:29.527925 5515 external_mini_cluster.cc:1428] Setting key 93b7b47d863e1721ed14b04bd30a1c4d
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[15707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263709, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:28:29.532833 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.523952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:60867 (local address 127.5.98.254:40761)
0924 15:28:29.523993 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.523995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.524198 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:28:29.524916 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.524918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.524932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.525001 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:29.526171 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.526703 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.527763 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.528067 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.528164 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.530264 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:29.530265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:29.530266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:29.530272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:29.531487 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.531948 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.531949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.531949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.532006 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.532225 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.532226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.532226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.532329 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:29.532404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.532636 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.532770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:28:29.532931 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.523920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40761 (local address 127.5.98.195:60867)
0924 15:28:29.524063 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.524107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.524778 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.525076 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.525080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.525822 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:28:29.526061 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.526067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.526807 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.526809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.527615 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.527617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.528176 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.528177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.528322 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.528832 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:28:29.528841 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:28:29.530179 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:28:29.531560 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.531566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.531567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.531878 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.532071 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:28:29.532071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:28:29.532072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:28:29.532167 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:28:29.532405 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:28:29.532409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:28:29.532515 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.532600 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.532751 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260924 15:28:29.533403 16211 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40761
I20260924 15:28:29.533476 16211 heartbeater.cc:464] Registering TS with master...
I20260924 15:28:29.533605 16211 heartbeater.cc:511] Master 127.5.98.254:40761 requested a full tablet report, sending...
I20260924 15:28:29.533948 15752 ts_manager.cc:194] Registered new tserver with Master: 17e0df97cd5b40c7a13f800de899dfe9 (127.5.98.195:43749)
I20260924 15:28:29.534264 15752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:60867
I20260924 15:28:29.539644 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:28:29.546694 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.540943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.5.98.254:40761)
0924 15:28:29.540996 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.540998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.541136 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:28:29.541189 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.541190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.541198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.541245 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:28:29.541780 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.542217 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.542924 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.543098 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.543144 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.544306 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:28:29.544307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:28:29.544307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:28:29.544313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:28:29.545741 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.546060 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.546061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.546061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.546116 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:28:29.546284 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:28:29.546285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:28:29.546285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:28:29.546400 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:28:29.546469 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.546577 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.546631 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:28:29.548286 15752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:28:29.549752 15752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:28:29.556483 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:33741 (local address 127.0.0.1:59010)
0924 15:28:29.552710 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.552720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.552766 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.553087 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.553089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.553091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:29.553133 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.553134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.554395 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.554396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.555481 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.555483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556277 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.556335 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556339 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.556383 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.556409 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260924 15:28:29.556505 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.5.98.195:43749)
0924 15:28:29.552254 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.552259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.552393 (+ 134us) server_negotiation.cc:407] Connection header received
0924 15:28:29.552509 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.552511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.552525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.552598 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:29.552819 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.553931 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.555318 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556057 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556203 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.556363 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.556408 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 15:28:29.556483 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:43749 (local address 127.0.0.1:47540)
0924 15:28:29.552278 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.552313 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.552413 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.552632 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.552635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.552637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:29.552704 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.552706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.554066 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.554067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.555223 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.555226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556155 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.556233 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556245 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.556339 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.556397 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":36,"thread_start_us":74,"threads_started":1}
I20260924 15:28:29.556757 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.5.98.194:33741)
0924 15:28:29.552871 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.552875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.552885 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:28:29.552924 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.552926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.552943 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.553016 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:29.553222 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.554258 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.555548 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556098 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556187 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556245 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.556627 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.556677 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260924 15:28:29.556983 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:38075 (local address 127.0.0.1:42530)
0924 15:28:29.552697 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.552707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.552906 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.553177 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.553178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.553179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:29.553212 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.553213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.554146 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.554147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.555398 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.555400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556741 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.556810 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556814 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.556869 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.556908 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":6,"thread_start_us":70,"threads_started":1}
I20260924 15:28:29.557320 16011 tablet_service.cc:1467] Processing CreateTablet for tablet f7b5c69d5e2640f599517940b0b615b3 (DEFAULT_TABLE table=test-table [id=f0954399bd2f413f88fba2ee25b84043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:29.557286 16145 tablet_service.cc:1467] Processing CreateTablet for tablet f7b5c69d5e2640f599517940b0b615b3 (DEFAULT_TABLE table=test-table [id=f0954399bd2f413f88fba2ee25b84043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:29.557473 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.552266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.5.98.193:38075)
0924 15:28:29.552446 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.552451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.552871 (+ 420us) server_negotiation.cc:407] Connection header received
0924 15:28:29.553001 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.553004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.553018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.553099 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:29.553266 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.554036 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.555503 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.556253 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.556351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.556408 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.557317 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.557390 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260924 15:28:29.557631 16011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b5c69d5e2640f599517940b0b615b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:29.557631 16145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b5c69d5e2640f599517940b0b615b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:29.558044 15877 tablet_service.cc:1467] Processing CreateTablet for tablet f7b5c69d5e2640f599517940b0b615b3 (DEFAULT_TABLE table=test-table [id=f0954399bd2f413f88fba2ee25b84043]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:28:29.558370 15877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b5c69d5e2640f599517940b0b615b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:28:29.560997 16228 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Bootstrap starting.
I20260924 15:28:29.561364 16230 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: Bootstrap starting.
I20260924 15:28:29.561652 16228 tablet_bootstrap.cc:654] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:29.561992 16230 tablet_bootstrap.cc:654] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:29.562187 16228 log.cc:824] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:29.562506 16230 log.cc:824] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:29.562501 16229 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Bootstrap starting.
I20260924 15:28:29.562947 16228 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: No bootstrap required, opened a new log
I20260924 15:28:29.563015 16228 ts_tablet_manager.cc:1397] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:29.563150 16229 tablet_bootstrap.cc:654] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Neither blocks nor log segments found. Creating new log.
I20260924 15:28:29.563261 16230 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: No bootstrap required, opened a new log
I20260924 15:28:29.563324 16230 ts_tablet_manager.cc:1397] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:28:29.563923 16229 log.cc:824] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:28:29.565304 16228 raft_consensus.cc:348] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.565467 16228 raft_consensus.cc:374] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:29.565385 16230 raft_consensus.cc:348] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.565507 16228 raft_consensus.cc:729] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a5879d2dd84d4da1136c72a6ceea4a, State: Initialized, Role: FOLLOWER
I20260924 15:28:29.565541 16230 raft_consensus.cc:374] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:29.565586 16230 raft_consensus.cc:729] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567d921ece9c4877b2311a8d42600675, State: Initialized, Role: FOLLOWER
I20260924 15:28:29.565651 16228 consensus_queue.cc:260] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.565701 16230 consensus_queue.cc:260] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.565824 16229 tablet_bootstrap.cc:492] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: No bootstrap required, opened a new log
I20260924 15:28:29.565884 16229 ts_tablet_manager.cc:1397] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:28:29.565946 16228 ts_tablet_manager.cc:1428] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:28:29.565968 16230 ts_tablet_manager.cc:1428] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:28:29.566072 16077 heartbeater.cc:503] Master 127.5.98.254:40761 was elected leader, sending a full tablet report...
I20260924 15:28:29.566074 15943 heartbeater.cc:503] Master 127.5.98.254:40761 was elected leader, sending a full tablet report...
I20260924 15:28:29.567798 16229 raft_consensus.cc:348] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.567930 16229 raft_consensus.cc:374] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:28:29.567982 16229 raft_consensus.cc:729] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e0df97cd5b40c7a13f800de899dfe9, State: Initialized, Role: FOLLOWER
I20260924 15:28:29.568097 16229 consensus_queue.cc:260] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.568338 16229 ts_tablet_manager.cc:1428] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:28:29.568390 16211 heartbeater.cc:503] Master 127.5.98.254:40761 was elected leader, sending a full tablet report...
W20260924 15:28:29.598717 16078 tablet.cc:2369] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:28:29.638684 16236 raft_consensus.cc:482] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:28:29.638813 16236 raft_consensus.cc:504] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.639199 16236 leader_election.cc:288] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 567d921ece9c4877b2311a8d42600675 (127.5.98.193:38075), d0a5879d2dd84d4da1136c72a6ceea4a (127.5.98.194:33741)
I20260924 15:28:29.643009 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.639357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:38075 (local address 127.5.98.195:56941)
0924 15:28:29.639394 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.639407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.639457 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.639637 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.639638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.639640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:29.639739 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.639742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.640752 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.640753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.641979 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.641982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.642740 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.642842 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.642844 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.642894 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.642935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":6}
I20260924 15:28:29.643013 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.639499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:43579 (local address 127.5.98.194:33741)
0924 15:28:29.639537 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.639539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.639829 (+ 290us) server_negotiation.cc:407] Connection header received
0924 15:28:29.639863 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.639863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.639877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.639945 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:29.640097 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.640790 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642077 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.642654 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642731 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.642739 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.642915 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.642957 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:28:29.643026 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.639434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:56941 (local address 127.5.98.193:38075)
0924 15:28:29.639483 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.639486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.639497 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:28:29.639529 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.639530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.639544 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.639606 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:28:29.639857 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.640664 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642078 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.642655 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642731 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.642739 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.642915 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.642959 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:28:29.643229 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.639419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:33741 (local address 127.5.98.195:43579)
0924 15:28:29.639683 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.639693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:28:29.639755 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:28:29.639969 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:28:29.639970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:28:29.639971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:28:29.640024 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.640025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.640861 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.640862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.641979 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:28:29.641982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.642740 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.642741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:28:29.642818 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.642823 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:28:29.643092 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 15:28:29.643137 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
I20260924 15:28:29.643397 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b5c69d5e2640f599517940b0b615b3" candidate_uuid: "17e0df97cd5b40c7a13f800de899dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567d921ece9c4877b2311a8d42600675" is_pre_election: true
I20260924 15:28:29.643600 15897 raft_consensus.cc:2454] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e0df97cd5b40c7a13f800de899dfe9 in term 0.
I20260924 15:28:29.643702 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b5c69d5e2640f599517940b0b615b3" candidate_uuid: "17e0df97cd5b40c7a13f800de899dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" is_pre_election: true
I20260924 15:28:29.643870 16031 raft_consensus.cc:2454] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17e0df97cd5b40c7a13f800de899dfe9 in term 0.
I20260924 15:28:29.643890 16099 leader_election.cc:302] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e0df97cd5b40c7a13f800de899dfe9, 567d921ece9c4877b2311a8d42600675; no voters:
I20260924 15:28:29.644011 16236 raft_consensus.cc:2781] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:28:29.644117 16236 raft_consensus.cc:482] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:28:29.644157 16236 raft_consensus.cc:3037] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:29.644870 16236 raft_consensus.cc:504] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.645015 16236 leader_election.cc:288] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [CANDIDATE]: Term 1 election: Requested vote from peers 567d921ece9c4877b2311a8d42600675 (127.5.98.193:38075), d0a5879d2dd84d4da1136c72a6ceea4a (127.5.98.194:33741)
I20260924 15:28:29.645147 16031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b5c69d5e2640f599517940b0b615b3" candidate_uuid: "17e0df97cd5b40c7a13f800de899dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a"
I20260924 15:28:29.645181 15897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b5c69d5e2640f599517940b0b615b3" candidate_uuid: "17e0df97cd5b40c7a13f800de899dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567d921ece9c4877b2311a8d42600675"
I20260924 15:28:29.645243 16031 raft_consensus.cc:3037] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:29.645274 15897 raft_consensus.cc:3037] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:28:29.645933 16031 raft_consensus.cc:2454] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e0df97cd5b40c7a13f800de899dfe9 in term 1.
I20260924 15:28:29.646009 15897 raft_consensus.cc:2454] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17e0df97cd5b40c7a13f800de899dfe9 in term 1.
I20260924 15:28:29.646126 16100 leader_election.cc:302] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17e0df97cd5b40c7a13f800de899dfe9, d0a5879d2dd84d4da1136c72a6ceea4a; no voters:
I20260924 15:28:29.646225 16236 raft_consensus.cc:2781] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:28:29.646363 16236 raft_consensus.cc:686] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 17e0df97cd5b40c7a13f800de899dfe9, State: Running, Role: LEADER
I20260924 15:28:29.646425 16236 consensus_queue.cc:237] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } }
I20260924 15:28:29.647198 15751 catalog_manager.cc:5495] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17e0df97cd5b40c7a13f800de899dfe9 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "17e0df97cd5b40c7a13f800de899dfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17e0df97cd5b40c7a13f800de899dfe9" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 } health_report { overall_health: UNKNOWN } } }
I20260924 15:28:29.670547 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.668245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.5.98.195:43749)
0924 15:28:29.668325 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 15:28:29.668327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:28:29.668334 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:28:29.668370 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:28:29.668371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:28:29.668388 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:28:29.668449 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:28:29.668563 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.669020 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.669832 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:28:29.670012 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:28:29.670061 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:28:29.670196 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:28:29.670298 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:28:29.670360 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:28:29.670392 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 15:28:29.670410 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 15:28:29.672516 15897 raft_consensus.cc:1261] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 1 FOLLOWER]: Refusing update from remote peer 17e0df97cd5b40c7a13f800de899dfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:29.672554 16031 raft_consensus.cc:1261] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 1 FOLLOWER]: Refusing update from remote peer 17e0df97cd5b40c7a13f800de899dfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:28:29.672892 16236 consensus_queue.cc:1035] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0a5879d2dd84d4da1136c72a6ceea4a" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:29.673033 16238 consensus_queue.cc:1035] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "567d921ece9c4877b2311a8d42600675" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 38075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:28:29.675385 16241 mvcc.cc:205] Tried to move back new op lower bound from 7332920154816016384 to 7332920154712289280. Current Snapshot: MvccSnapshot[applied={T|T < 7332920154816016384}]
I20260924 15:28:29.676335 16243 mvcc.cc:205] Tried to move back new op lower bound from 7332920154816016384 to 7332920154712289280. Current Snapshot: MvccSnapshot[applied={T|T < 7332920154816016384}]
I20260924 15:28:29.675515 16242 mvcc.cc:205] Tried to move back new op lower bound from 7332920154816016384 to 7332920154712289280. Current Snapshot: MvccSnapshot[applied={T|T < 7332920154816016384}]
I20260924 15:28:29.678185 15751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:29.678287 15751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:28:29.679087 15751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20264369b4ea4466b3471f70f1aec286: Sending DeleteTablet for 3 replicas of tablet f7b5c69d5e2640f599517940b0b615b3
I20260924 15:28:29.679407 16145 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b5c69d5e2640f599517940b0b615b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:47540
I20260924 15:28:29.679504 15877 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b5c69d5e2640f599517940b0b615b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42530
I20260924 15:28:29.679512 16011 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b5c69d5e2640f599517940b0b615b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:28:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:59010
I20260924 15:28:29.679754 16251 tablet_replica.cc:330] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: stopping tablet replica
I20260924 15:28:29.679752 16253 tablet_replica.cc:330] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: stopping tablet replica
I20260924 15:28:29.679843 16251 raft_consensus.cc:2229] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:28:29.679849 16253 raft_consensus.cc:2229] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:29.679934 16251 raft_consensus.cc:2258] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:29.679943 16253 raft_consensus.cc:2258] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:29.680166 16252 tablet_replica.cc:330] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675: stopping tablet replica
I20260924 15:28:29.680296 16252 raft_consensus.cc:2229] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:28:29.680306 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15814
I20260924 15:28:29.680451 16252 raft_consensus.cc:2258] T f7b5c69d5e2640f599517940b0b615b3 P 567d921ece9c4877b2311a8d42600675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:28:29.680492 16253 ts_tablet_manager.cc:1905] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:29.682188 16253 ts_tablet_manager.cc:1918] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:29.682264 16253 log.cc:1196] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/f7b5c69d5e2640f599517940b0b615b3
I20260924 15:28:29.682549 16253 ts_tablet_manager.cc:1939] T f7b5c69d5e2640f599517940b0b615b3 P d0a5879d2dd84d4da1136c72a6ceea4a: Deleting consensus metadata
I20260924 15:28:29.683127 15738 catalog_manager.cc:4841] TS d0a5879d2dd84d4da1136c72a6ceea4a (127.5.98.194:33741): tablet f7b5c69d5e2640f599517940b0b615b3 (table test-table [id=f0954399bd2f413f88fba2ee25b84043]) successfully deleted
W20260924 15:28:29.687549 15737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:38075 (error 108)
I20260924 15:28:29.688087 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:28:29.687935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:38075 (local address 127.0.0.1:42532)
0924 15:28:29.687996 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:28:29.688040 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:38075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:28:29.688141 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15948
I20260924 15:28:29.688331 16251 ts_tablet_manager.cc:1905] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:28:29.690412 16251 ts_tablet_manager.cc:1918] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:28:29.691699 16251 log.cc:1196] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/f7b5c69d5e2640f599517940b0b615b3
I20260924 15:28:29.692001 16251 ts_tablet_manager.cc:1939] T f7b5c69d5e2640f599517940b0b615b3 P 17e0df97cd5b40c7a13f800de899dfe9: Deleting consensus metadata
I20260924 15:28:29.692427 15736 catalog_manager.cc:4841] TS 17e0df97cd5b40c7a13f800de899dfe9 (127.5.98.195:43749): tablet f7b5c69d5e2640f599517940b0b615b3 (table test-table [id=f0954399bd2f413f88fba2ee25b84043]) successfully deleted
W20260924 15:28:29.693890 15737 catalog_manager.cc:4584] TS 567d921ece9c4877b2311a8d42600675 (127.5.98.193:38075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7b5c69d5e2640f599517940b0b615b3: Network error: Client connection negotiation failed: client connection to 127.5.98.193:38075: connect: Connection refused (error 111)
I20260924 15:28:29.694281 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 16082
I20260924 15:28:29.699347 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 15720
2026-09-24T15:28:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2019 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[16257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[16257](info): set up 2 sockets
Sep 24 15:28:29 dist-test-slave-4g6f krb5kdc[16257](info): commencing operation
krb5kdc: starting...
W20260924 15:28:30.745687 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:28:30 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:28:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangeradmin_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangeradmin_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangerlookup_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangerlookup_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
I20260924 15:28:30.900017 5515 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskC7bDtI/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/postgres -l logfile start
2026-09-24 15:28:33.506 UTC [16287] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:28:33.506 UTC [16287] LOG: listening on IPv4 address "127.5.98.212", port 45683
2026-09-24 15:28:33.509 UTC [16287] LOG: listening on Unix socket "/tmp/.s.PGSQL.45683"
2026-09-24 15:28:33.531 UTC [16290] LOG: database system was shut down at 2026-09-24 15:28:32 UTC
2026-09-24 15:28:33.535 UTC [16287] LOG: database system is ready to accept connections
I20260924 15:28:34.513578 5515 mini_postgres.cc:96] Postgres bound to 45683
2026-09-24 15:28:34.521 UTC [16298] FATAL: database "slave" does not exist
127.5.98.212:45683 - accepting connections
I20260924 15:28:34.521746 5515 mini_ranger.cc:161] Starting Ranger...
I20260924 15:28:34.536764 5515 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:28:34.999969 5515 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:28:35.000062 5515 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin
2026-09-24 15:28:35,424 [I] DB FLAVOR :POSTGRES
2026-09-24 15:28:35,425 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:28:35,425 [I] Checking connection..
2026-09-24 15:28:35,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:28:35,699 [I] Checking connection passed.
2026-09-24 15:28:35,699 [I] --------- Verifying version history table ---------
2026-09-24 15:28:35,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:28:35,925 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:28:35,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:28:36,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:28:36,174 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:28:36,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:28:36.405 UTC [16362] WARNING: there is no transaction in progress
2026-09-24 15:28:36,412 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:28:36,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:28:36,652 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:28:36,652 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:28:36,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:28:36,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:28:37,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4g6f.c.gcp-upstream.internal','N') ;"
2026-09-24 15:28:37,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:28:37,620 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:28:37,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:28:37,868 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:28:37,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:28:38,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:28:38,366 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:28:38,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:28:39.336 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.359 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.381 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.403 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.432 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.464 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.547 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.557 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.576 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.591 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.606 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.624 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.637 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.645 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.653 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:39.661 UTC [16470] WARNING: there is no transaction in progress
2026-09-24 15:28:40,304 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:28:40,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:28:40,551 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:28:40,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:28:40,799 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:28:40,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:28:41,061 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:28:41,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:28:41,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:28:41,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4g6f.c.gcp-upstream.internal';"
2026-09-24 15:28:41,810 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:28:41,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:28:42,032 [I] DB_PATCHES have already been applied
I20260924 15:28:42.038110 5515 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskC7bDtI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:28:42.038175 5515 mini_ranger.cc:193] Using host: 127.5.98.212
I20260924 15:28:42.041366 5515 mini_ranger.cc:227] Ranger admin URL: http://127.5.98.212:37615
Sep 24, 2026 3:28:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:28:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:28:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:28:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:28:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:28:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:28:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:28:43.154378 5515 mini_ranger.cc:160] Time spent starting Ranger: real 8.633s user 0.000s sys 0.008s
Sep 24, 2026 3:29:04 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:29:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:29:04 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:29:04 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:29:05 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:29:06.228914 5515 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangerkms_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/rangerkms_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.98.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.98.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:29:06 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263746, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:29:06.311910 5515 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:29:06.321448 5515 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:29:07.172672 5515 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:29:07.172760 5515 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms
I20260924 15:29:07.172783 5515 mini_ranger_kms.cc:227] Using postgres at 127.5.98.212:45683
2026-09-24 15:29:07,793 [I] DB FLAVOR :POSTGRES
2026-09-24 15:29:07,793 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:29:07,793 [I] Checking connection
2026-09-24 15:29:07,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:29:08,058 [I] connection success
2026-09-24 15:29:08,059 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:29:08,059 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:29:08,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:29:08,331 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:29:08,331 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:29:08,331 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:29:08,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:45683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:29:08,685 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:29:09.208690 5515 mini_ranger_kms.cc:310] Created kms service
I20260924 15:29:09.409422 5515 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:29:09.454763 5515 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:29:09 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263746, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:09.858965 5515 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:29:09.859114 5515 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms:/tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:29:09.859136 5515 mini_ranger_kms.cc:240] Using host: 127.5.98.212
I20260924 15:29:09.862548 5515 mini_ranger_kms.cc:276] Ranger KMS PID: 16651
I20260924 15:29:09.862679 5515 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.98.212:39835
Sep 24, 2026 3:29:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:29:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:29:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:29:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:29:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:29:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 15:29:12.135254 5515 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.823s user 0.000s sys 0.314s
I20260924 15:29:12.135363 5515 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:29:12.135471 5515 mini_ranger_kms.cc:401] 127.5.98.212:39835/kms/v1/keys
Sep 24 15:29:12 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 15:29:12 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
Sep 24 15:29:12 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
Sep 24 15:29:15 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:29:15.095634 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40057
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:40057
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39835/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:29:15.182854 16701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:29:15.183077 16701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:29:15.183099 16701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:29:15.184319 16701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:29:15.184367 16701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:29:15.184381 16701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:29:15.184394 16701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:29:15.184406 16701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:29:15.185943 16701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.98.212:39835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:40057
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:29:15.186210 16701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:29:15.186462 16701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:29:15.192477 16707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:15.192467 16706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:15.192564 16709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:29:15.192775 16701 server_base.cc:1034] running on GCE node
I20260924 15:29:15.193095 16701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:29:15.193447 16701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:29:15.194633 16701 hybrid_clock.cc:648] HybridClock initialized: now 1790263755194612 us; error 45 us; skew 500 ppm
Sep 24 15:29:15 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:29:15.197517 16701 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:29:15.197988 16701 webserver.cc:460] Webserver started at http://127.5.98.254:37361/ using document root <none> and password file <none>
I20260924 15:29:15.198230 16701 fs_manager.cc:359] Metadata directory not provided
I20260924 15:29:15.198295 16701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:29:15.198452 16701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:29:15 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:15.242864 16701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "540332d4bea248f7b918c31fcd808292"
format_stamp: "Formatted at 2026-09-24 15:29:15 on dist-test-slave-4g6f"
server_key: "e64ad5dcf8a4cb9067ca44bce3559195"
server_key_iv: "37ecb9b4f248524b798f123d2f92c78f"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:15.243316 16701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "540332d4bea248f7b918c31fcd808292"
format_stamp: "Formatted at 2026-09-24 15:29:15 on dist-test-slave-4g6f"
server_key: "e64ad5dcf8a4cb9067ca44bce3559195"
server_key_iv: "37ecb9b4f248524b798f123d2f92c78f"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:15.244894 16701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:29:15.259588 16716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:15.261363 16701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:29:15.261480 16701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "540332d4bea248f7b918c31fcd808292"
format_stamp: "Formatted at 2026-09-24 15:29:15 on dist-test-slave-4g6f"
server_key: "e64ad5dcf8a4cb9067ca44bce3559195"
server_key_iv: "37ecb9b4f248524b798f123d2f92c78f"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:15.261605 16701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:29:15.273664 16701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:29:15.276386 16701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:29:15.276561 16701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:29:15.281411 16701 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:40057
I20260924 15:29:15.281512 16769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:40057 every 8 connection(s)
I20260924 15:29:15.281867 16701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:29:15.282549 16770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:29:15.284859 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: Bootstrap starting.
I20260924 15:29:15.285468 16770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: Neither blocks nor log segments found. Creating new log.
I20260924 15:29:15.285784 16770 log.cc:824] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: Log is configured to *not* fsync() on all Append() calls
I20260924 15:29:15.286450 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: No bootstrap required, opened a new log
I20260924 15:29:15.288584 16770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } }
I20260924 15:29:15.288730 16770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:29:15.288753 16770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540332d4bea248f7b918c31fcd808292, State: Initialized, Role: FOLLOWER
I20260924 15:29:15.288905 16770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } }
I20260924 15:29:15.288978 16770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:29:15.289000 16770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:29:15.289074 16770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:29:15.289595 16770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } }
I20260924 15:29:15.289707 16770 leader_election.cc:302] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540332d4bea248f7b918c31fcd808292; no voters:
I20260924 15:29:15.289898 16770 leader_election.cc:288] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:29:15.289961 16773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:29:15.290167 16770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:29:15.290184 16773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [term 1 LEADER]: Becoming Leader. State: Replica: 540332d4bea248f7b918c31fcd808292, State: Running, Role: LEADER
I20260924 15:29:15.291558 16773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } }
I20260924 15:29:15.291632 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 16701
I20260924 15:29:15.291745 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:29:15.291940 5515 external_mini_cluster.cc:1428] Setting key cc60fff6d28ee1ba4de06e96c97fbbbf
I20260924 15:29:15.292484 16774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540332d4bea248f7b918c31fcd808292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } } }
I20260924 15:29:15.292569 16774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [sys.catalog]: This master's current role is: LEADER
I20260924 15:29:15.292768 16775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540332d4bea248f7b918c31fcd808292. Latest consensus state: current_term: 1 leader_uuid: "540332d4bea248f7b918c31fcd808292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540332d4bea248f7b918c31fcd808292" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40057 } } }
I20260924 15:29:15.292788 16789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:29:15.292819 16775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292 [sys.catalog]: This master's current role is: LEADER
I20260924 15:29:15.293309 16789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:29:15.293295 16770 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:29:15.304445 16789 catalog_manager.cc:1347] Generated new cluster ID: 63ef43d63e374edfaa6a48e16c42b265
I20260924 15:29:15.304641 16789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:29:15 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:29:15.318660 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:15.292608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.5.98.254:40057)
0924 15:29:15.294094 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:15.294104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:15.295912 (+ 1808us) server_negotiation.cc:407] Connection header received
0924 15:29:15.296064 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:15.296072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:15.296136 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:15.296290 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:29:15.297077 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:15.303917 (+ 6840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:15.310518 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:15.311188 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:15.311301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:15.314482 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:29:15.314486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:29:15.314487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:29:15.314499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:29:15.316603 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:15.316988 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:15.316991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:15.316991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:15.317092 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:15.317263 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:15.317264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:15.317264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:15.317562 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:29:15.318104 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:15.318174 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:29:15.318296 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":96,"threads_started":1}
I20260924 15:29:15.332967 16789 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:29:15.333808 16789 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:29:15.339072 16789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: Generated new TSK 0
I20260924 15:29:15.339319 16789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:29:16.158445 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:29:16.290645 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:29:16.290836 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:29:16.290859 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:29:16.292186 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:29:16.292227 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:29:16.292254 16815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:29:16.293684 16815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.98.212:39835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:29:16.293903 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:29:16.294157 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:29:16.299309 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:16.299278 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:29:16.299651 16815 server_base.cc:1034] running on GCE node
W20260924 15:29:16.299315 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:29:16.300038 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:29:16.300315 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:29:16.301477 16815 hybrid_clock.cc:648] HybridClock initialized: now 1790263756301466 us; error 28 us; skew 500 ppm
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:29:16.307510 16815 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:29:16.308065 16815 webserver.cc:460] Webserver started at http://127.5.98.193:35061/ using document root <none> and password file <none>
I20260924 15:29:16.308362 16815 fs_manager.cc:359] Metadata directory not provided
I20260924 15:29:16.308444 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:29:16.308593 16815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:16.321394 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "d86689e0a3474ef6bff98257a901f05e"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "fd73555220ff1291e995df1ad88aadb0"
server_key_iv: "8ec80fdc17d516e737ce1fbb2b6461c0"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.321975 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "d86689e0a3474ef6bff98257a901f05e"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "fd73555220ff1291e995df1ad88aadb0"
server_key_iv: "8ec80fdc17d516e737ce1fbb2b6461c0"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.323510 16815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:29:16.335883 16833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.339810 16815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 15:29:16.339933 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "d86689e0a3474ef6bff98257a901f05e"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "fd73555220ff1291e995df1ad88aadb0"
server_key_iv: "8ec80fdc17d516e737ce1fbb2b6461c0"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.340022 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:29:16.352710 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:29:16.354801 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:29:16.354979 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:29:16.355288 16815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:29:16.355695 16815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:29:16.355741 16815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.355775 16815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:29:16.355834 16815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.363117 16815 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:33317
I20260924 15:29:16.363222 16946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:33317 every 8 connection(s)
I20260924 15:29:16.363526 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:29:16.365291 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 16815
I20260924 15:29:16.365402 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:29:16.365620 5515 external_mini_cluster.cc:1428] Setting key d7597f780ad538bbc3bff530f2a0879a
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:29:16.374522 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.364266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40057 (local address 127.5.98.193:58811)
0924 15:29:16.364434 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.364476 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.364981 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.365285 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.365290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.365665 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:29:16.365966 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.365973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.366792 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.366793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.368100 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.368103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.368517 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.368519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.368645 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.369204 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:29:16.369214 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:29:16.370819 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:29:16.372881 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.372888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.372890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.373175 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.373419 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.373420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.373421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.373553 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.373854 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:29:16.373860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:29:16.374027 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 15:29:16.374138 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:29:16.374288 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260924 15:29:16.374972 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.364307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:58811 (local address 127.5.98.254:40057)
0924 15:29:16.364434 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.364436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.364579 (+ 143us) server_negotiation.cc:407] Connection header received
0924 15:29:16.365091 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.365093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.365108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.365197 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:29:16.366050 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.366665 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.368189 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.368416 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.368506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.370937 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:29:16.370939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:29:16.370939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:29:16.370946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:29:16.372789 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.373286 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.373287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.373287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.373349 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.373629 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.373630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.373630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.373764 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:29:16.373827 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:16.374770 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 15:29:16.374862 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260924 15:29:16.375538 16947 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40057
I20260924 15:29:16.375698 16947 heartbeater.cc:464] Registering TS with master...
I20260924 15:29:16.375902 16947 heartbeater.cc:511] Master 127.5.98.254:40057 requested a full tablet report, sending...
I20260924 15:29:16.376488 16733 ts_manager.cc:194] Registered new tserver with Master: d86689e0a3474ef6bff98257a901f05e (127.5.98.193:33317)
I20260924 15:29:16.377405 16733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:58811
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:29:16.411537 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 15:29:16.536605 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:29:16.536907 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:29:16.536947 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:29:16.538808 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:29:16.538868 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:29:16.538918 16955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:29:16.541244 16955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.98.212:39835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:29:16.541489 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:29:16.541762 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:29:16.547511 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:16.547538 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:16.547541 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:29:16.548040 16955 server_base.cc:1034] running on GCE node
I20260924 15:29:16.548283 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:29:16.548542 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:29:16.549705 16955 hybrid_clock.cc:648] HybridClock initialized: now 1790263756549683 us; error 37 us; skew 500 ppm
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:29:16.552716 16955 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:29:16.553182 16955 webserver.cc:460] Webserver started at http://127.5.98.194:34767/ using document root <none> and password file <none>
I20260924 15:29:16.553426 16955 fs_manager.cc:359] Metadata directory not provided
I20260924 15:29:16.553524 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:29:16.553673 16955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:16.567610 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "e4f8a18d51b54d15b13830b80ea0ba08"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "c95bdb9b9eb2002f34c3066fefbe2d15"
server_key_iv: "d51bd32bc098f1572b93db2c28d87b5d"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.568264 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "e4f8a18d51b54d15b13830b80ea0ba08"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "c95bdb9b9eb2002f34c3066fefbe2d15"
server_key_iv: "d51bd32bc098f1572b93db2c28d87b5d"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.570325 16955 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:29:16.587651 16971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.587951 16955 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 15:29:16.588109 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "e4f8a18d51b54d15b13830b80ea0ba08"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "c95bdb9b9eb2002f34c3066fefbe2d15"
server_key_iv: "d51bd32bc098f1572b93db2c28d87b5d"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.588291 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:29:16.612920 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:29:16.615648 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:29:16.615900 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:29:16.616163 16955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:29:16.616601 16955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:29:16.616695 16955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.616775 16955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:29:16.616842 16955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.623240 16955 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:42439
I20260924 15:29:16.623790 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:29:16.626776 17084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:42439 every 8 connection(s)
I20260924 15:29:16.628096 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 16955
I20260924 15:29:16.628185 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:29:16.628412 5515 external_mini_cluster.cc:1428] Setting key e371f1b1b4982a051ee92c45c594073f
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:29:16.641036 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.627481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:34851 (local address 127.5.98.254:40057)
0924 15:29:16.627530 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.627533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.627784 (+ 251us) server_negotiation.cc:407] Connection header received
0924 15:29:16.631230 (+ 3446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.631231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.631246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.631275 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:29:16.632194 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.632995 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.634513 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.634827 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.634869 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.637590 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:29:16.637592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:29:16.637592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:29:16.637600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:29:16.639334 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.639876 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.639878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.639878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.639937 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.640220 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.640221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.640221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.640385 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:29:16.640444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:16.640843 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 15:29:16.640934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:29:16.641968 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.627479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40057 (local address 127.5.98.194:34851)
0924 15:29:16.627628 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.627704 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.631097 (+ 3393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.631471 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.631475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.631740 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:29:16.632074 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.632081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.633117 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.633118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.634390 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.634393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.634961 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.634962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.635125 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.635787 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:29:16.635798 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:29:16.637474 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:29:16.639435 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.639442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.639445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.639784 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.640005 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.640005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.640006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.640138 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.640520 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:29:16.640527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:29:16.640688 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:29:16.641572 (+ 884us) client_negotiation.cc:240] Negotiation successful
0924 15:29:16.641739 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260924 15:29:16.642661 17085 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40057
I20260924 15:29:16.642763 17085 heartbeater.cc:464] Registering TS with master...
I20260924 15:29:16.642968 17085 heartbeater.cc:511] Master 127.5.98.254:40057 requested a full tablet report, sending...
I20260924 15:29:16.643392 16733 ts_manager.cc:194] Registered new tserver with Master: e4f8a18d51b54d15b13830b80ea0ba08 (127.5.98.194:42439)
I20260924 15:29:16.643909 16733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:34851
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:29:16.674685 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:29:16.814211 17090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:29:16.814499 17090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:29:16.814541 17090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:29:16.815821 17090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:29:16.815882 17090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:29:16.815925 17090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:29:16.817399 17090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.98.212:39835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:29:16.817648 17090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:29:16.817869 17090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:29:16.823483 17098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:16.823591 17095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:29:16.823499 17096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:29:16.823859 17090 server_base.cc:1034] running on GCE node
I20260924 15:29:16.826799 17090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:29:16.827080 17090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:29:16.828260 17090 hybrid_clock.cc:648] HybridClock initialized: now 1790263756828247 us; error 31 us; skew 500 ppm
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:29:16.831022 17090 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:29:16.831455 17090 webserver.cc:460] Webserver started at http://127.5.98.195:36683/ using document root <none> and password file <none>
I20260924 15:29:16.831789 17090 fs_manager.cc:359] Metadata directory not provided
I20260924 15:29:16.831861 17090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:29:16.831971 17090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:16.843650 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "093956dee3f443e48ea11ef51d367f2f"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "eec32b5b84c682be5753b683755e6cf4"
server_key_iv: "e8bd5f3f04252e4439cb63d910402172"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.844188 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "093956dee3f443e48ea11ef51d367f2f"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "eec32b5b84c682be5753b683755e6cf4"
server_key_iv: "e8bd5f3f04252e4439cb63d910402172"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.847245 17090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 15:29:16.871099 17105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.875869 17090 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.002s
I20260924 15:29:16.876006 17090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "093956dee3f443e48ea11ef51d367f2f"
format_stamp: "Formatted at 2026-09-24 15:29:16 on dist-test-slave-4g6f"
server_key: "eec32b5b84c682be5753b683755e6cf4"
server_key_iv: "e8bd5f3f04252e4439cb63d910402172"
server_key_version: "kuduclusterkey@0"
I20260924 15:29:16.876120 17090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:29:16.900432 17090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:29:16.903067 17090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:29:16.903285 17090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:29:16.903839 17090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:29:16.904390 17090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:29:16.904490 17090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.904662 17090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:29:16.904727 17090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:29:16.915089 17090 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:40611
I20260924 15:29:16.918476 17218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:40611 every 8 connection(s)
I20260924 15:29:16.918978 17090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:29:16.919569 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 17090
I20260924 15:29:16.919762 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:29:16.920028 5515 external_mini_cluster.cc:1428] Setting key c4e90171aeeca8947d799ca95f7446de
Sep 24 15:29:16 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:29:16.941642 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.921929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:33701 (local address 127.5.98.254:40057)
0924 15:29:16.921975 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.921977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.927957 (+ 5980us) server_negotiation.cc:407] Connection header received
0924 15:29:16.928648 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.928650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.928664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.928749 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:29:16.929503 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.930296 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.931898 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.932200 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.932276 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.934969 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:29:16.934970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:29:16.934970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:29:16.934979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:29:16.936921 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.940081 (+ 3160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.940083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.940083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.940151 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.940464 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.940465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.940465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.940611 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:29:16.940666 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:16.941442 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 15:29:16.941542 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:29:16.942492 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.921920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:40057 (local address 127.5.98.195:33701)
0924 15:29:16.927804 (+ 5884us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.927861 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.928538 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.928831 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.928835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.929100 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:29:16.929396 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.929403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.930521 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.930522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.931817 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.931820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.932295 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.932296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.932461 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.933065 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:29:16.933076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:29:16.934863 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:29:16.939335 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.939345 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.939348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.939983 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.940224 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:29:16.940225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:29:16.940225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:29:16.940375 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:29:16.941095 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:29:16.941102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:29:16.941273 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 15:29:16.942025 (+ 752us) client_negotiation.cc:240] Negotiation successful
0924 15:29:16.942216 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5825,"thread_start_us":105,"threads_started":1}
I20260924 15:29:16.943785 17219 heartbeater.cc:347] Connected to a master server at 127.5.98.254:40057
I20260924 15:29:16.943888 17219 heartbeater.cc:464] Registering TS with master...
I20260924 15:29:16.944089 17219 heartbeater.cc:511] Master 127.5.98.254:40057 requested a full tablet report, sending...
I20260924 15:29:16.944509 16733 ts_manager.cc:194] Registered new tserver with Master: 093956dee3f443e48ea11ef51d367f2f (127.5.98.195:40611)
I20260924 15:29:16.945015 16733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:33701
I20260924 15:29:16.953066 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:29:16.972074 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.956183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.5.98.254:40057)
0924 15:29:16.956225 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.956227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.959857 (+ 3630us) server_negotiation.cc:407] Connection header received
0924 15:29:16.959965 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.959967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.959978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.960007 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:29:16.960803 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.961568 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.963423 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.963753 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.963801 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.965755 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:29:16.965757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:29:16.965757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:29:16.965764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:29:16.968355 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.970400 (+ 2045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.970401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.970401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.970472 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:29:16.970655 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:29:16.970656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:29:16.970656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:29:16.970770 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:29:16.970836 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:16.971903 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 15:29:16.971982 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:29:16.978053 16733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:29:16.980489 16733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:29:16.993594 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.984544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33317 (local address 127.0.0.1:56480)
0924 15:29:16.985925 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.985934 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.985972 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.986354 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.986357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.986360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:29:16.986435 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.986437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.988258 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.988259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.990315 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.990319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.991910 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.991912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:16.992000 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.992031 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 15:29:16.992173 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 15:29:16.993270 (+ 1097us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":154,"threads_started":1}
I20260924 15:29:16.994338 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.984890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.5.98.193:33317)
0924 15:29:16.986085 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.986090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.986101 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:29:16.986139 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.986141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.986158 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.986233 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:29:16.986526 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.988144 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.990447 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.991620 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.991793 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:16.991879 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:16.994162 (+ 2283us) server_negotiation.cc:299] Negotiation successful
0924 15:29:16.994215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":59,"threads_started":1}
I20260924 15:29:16.996554 16881 tablet_service.cc:1467] Processing CreateTablet for tablet 52b1dad2a7124b08914733cb2726ae58 (DEFAULT_TABLE table=test-table [id=28ab0a2b6f7b4de2953502cba2c1ad86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:29:16.996968 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b1dad2a7124b08914733cb2726ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:29:17.002715 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.985147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:42439 (local address 127.0.0.1:44178)
0924 15:29:16.989838 (+ 4691us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.989851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.989904 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.992349 (+ 2445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.992350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.992353 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:29:16.992870 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.992873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.999063 (+ 6190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:16.999064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.000729 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.000732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.002460 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.002461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.002525 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.002529 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:29:17.002586 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:29:17.002630 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4642,"mutex_wait_us":415,"thread_start_us":121,"threads_started":1}
I20260924 15:29:17.005785 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.985319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.5.98.194:42439)
0924 15:29:16.992004 (+ 6685us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.992010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.992023 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:29:16.992066 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.992069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.992086 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.992182 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:29:16.994596 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.998958 (+ 4362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.000948 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.002202 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.002284 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.002419 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:17.005599 (+ 3180us) server_negotiation.cc:299] Negotiation successful
0924 15:29:17.005647 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6616,"thread_start_us":129,"threads_started":1}
I20260924 15:29:17.006794 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.984248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:40611 (local address 127.0.0.1:55070)
0924 15:29:16.985622 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:16.985666 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:16.985735 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:16.996702 (+ 10967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:16.996703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:16.996707 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:29:16.996864 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:16.996867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.003128 (+ 6261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.003129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.004371 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.004375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.006592 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.006594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.006655 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.006659 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:29:17.006686 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 15:29:17.006725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":136,"threads_started":1}
I20260924 15:29:17.006934 17019 tablet_service.cc:1467] Processing CreateTablet for tablet 52b1dad2a7124b08914733cb2726ae58 (DEFAULT_TABLE table=test-table [id=28ab0a2b6f7b4de2953502cba2c1ad86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:29:17.007342 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:16.984380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.5.98.195:40611)
0924 15:29:16.996474 (+ 12094us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:16.996480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:16.996492 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:29:16.996555 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:16.996558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:16.996576 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:16.996970 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:29:16.997334 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:16.998651 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.004600 (+ 5949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.006495 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.007103 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.007197 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:17.007223 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 15:29:17.007265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12027,"thread_start_us":84,"threads_started":1}
I20260924 15:29:17.007404 17019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b1dad2a7124b08914733cb2726ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:29:17.007952 17153 tablet_service.cc:1467] Processing CreateTablet for tablet 52b1dad2a7124b08914733cb2726ae58 (DEFAULT_TABLE table=test-table [id=28ab0a2b6f7b4de2953502cba2c1ad86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:29:17.008251 17153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52b1dad2a7124b08914733cb2726ae58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:29:17.011524 17239 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Bootstrap starting.
I20260924 15:29:17.012857 17239 tablet_bootstrap.cc:654] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Neither blocks nor log segments found. Creating new log.
I20260924 15:29:17.013228 17239 log.cc:824] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:29:17.013984 17239 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: No bootstrap required, opened a new log
I20260924 15:29:17.014043 17239 ts_tablet_manager.cc:1397] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:29:17.015977 17239 raft_consensus.cc:348] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.016134 17239 raft_consensus.cc:374] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:29:17.016163 17239 raft_consensus.cc:729] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093956dee3f443e48ea11ef51d367f2f, State: Initialized, Role: FOLLOWER
I20260924 15:29:17.016268 17239 consensus_queue.cc:260] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.016494 17239 ts_tablet_manager.cc:1428] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:29:17.016621 17219 heartbeater.cc:503] Master 127.5.98.254:40057 was elected leader, sending a full tablet report...
I20260924 15:29:17.019836 17238 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: Bootstrap starting.
I20260924 15:29:17.019991 17240 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Bootstrap starting.
I20260924 15:29:17.020726 17238 tablet_bootstrap.cc:654] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: Neither blocks nor log segments found. Creating new log.
I20260924 15:29:17.020923 17240 tablet_bootstrap.cc:654] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Neither blocks nor log segments found. Creating new log.
I20260924 15:29:17.021167 17238 log.cc:824] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:29:17.021266 17240 log.cc:824] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Log is configured to *not* fsync() on all Append() calls
I20260924 15:29:17.024497 17238 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: No bootstrap required, opened a new log
I20260924 15:29:17.024577 17240 tablet_bootstrap.cc:492] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: No bootstrap required, opened a new log
I20260924 15:29:17.024585 17238 ts_tablet_manager.cc:1397] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 15:29:17.024637 17240 ts_tablet_manager.cc:1397] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 15:29:17.026572 17240 raft_consensus.cc:348] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.026731 17240 raft_consensus.cc:374] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:29:17.026759 17240 raft_consensus.cc:729] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f8a18d51b54d15b13830b80ea0ba08, State: Initialized, Role: FOLLOWER
I20260924 15:29:17.026875 17240 consensus_queue.cc:260] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.027120 17240 ts_tablet_manager.cc:1428] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:29:17.027328 17085 heartbeater.cc:503] Master 127.5.98.254:40057 was elected leader, sending a full tablet report...
I20260924 15:29:17.027916 17238 raft_consensus.cc:348] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.028153 17238 raft_consensus.cc:374] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:29:17.028251 17238 raft_consensus.cc:729] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86689e0a3474ef6bff98257a901f05e, State: Initialized, Role: FOLLOWER
I20260924 15:29:17.028484 17238 consensus_queue.cc:260] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.028856 17238 ts_tablet_manager.cc:1428] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 15:29:17.029024 16947 heartbeater.cc:503] Master 127.5.98.254:40057 was elected leader, sending a full tablet report...
I20260924 15:29:17.033191 17245 raft_consensus.cc:482] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:29:17.033350 17245 raft_consensus.cc:504] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.038026 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.033908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:40611 (local address 127.5.98.194:54755)
0924 15:29:17.034012 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:17.034024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:17.034068 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:17.034344 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:17.034345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:17.034347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:29:17.034421 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.034423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.035503 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.035505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.036861 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.036864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.037735 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.037736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.037827 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.037831 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:29:17.037900 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:29:17.037945 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 15:29:17.038508 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.034154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:54755 (local address 127.5.98.195:40611)
0924 15:29:17.034198 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:17.034200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:17.034208 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:29:17.034234 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:17.034235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:17.034244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:17.034293 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:29:17.034503 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.035341 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.036974 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.037585 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.037666 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.037676 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:17.038364 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 15:29:17.038408 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:29:17.038800 17173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b1dad2a7124b08914733cb2726ae58" candidate_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093956dee3f443e48ea11ef51d367f2f" is_pre_election: true
I20260924 15:29:17.038960 17173 raft_consensus.cc:2454] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f8a18d51b54d15b13830b80ea0ba08 in term 0.
I20260924 15:29:17.039384 16973 leader_election.cc:302] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093956dee3f443e48ea11ef51d367f2f, e4f8a18d51b54d15b13830b80ea0ba08; no voters:
I20260924 15:29:17.043774 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.039870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33317 (local address 127.5.98.194:39969)
0924 15:29:17.040039 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:17.040050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:29:17.040095 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:29:17.040273 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:29:17.040274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:29:17.040275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:29:17.040307 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.040308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.041456 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.041457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.042622 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:29:17.042625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.043480 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.043481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:29:17.043585 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.043589 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:29:17.043646 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:29:17.043703 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 15:29:17.044152 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.039965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:39969 (local address 127.5.98.193:33317)
0924 15:29:17.040010 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:17.040012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:17.040150 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:29:17.040176 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:17.040177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:17.040187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:17.040232 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:29:17.040382 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.041138 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.042728 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.043351 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.043432 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.043443 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:17.044015 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 15:29:17.044085 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:29:17.044660 17245 leader_election.cc:288] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093956dee3f443e48ea11ef51d367f2f (127.5.98.195:40611), d86689e0a3474ef6bff98257a901f05e (127.5.98.193:33317)
I20260924 15:29:17.044708 16901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b1dad2a7124b08914733cb2726ae58" candidate_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86689e0a3474ef6bff98257a901f05e" is_pre_election: true
I20260924 15:29:17.044866 16901 raft_consensus.cc:2454] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4f8a18d51b54d15b13830b80ea0ba08 in term 0.
I20260924 15:29:17.044785 17245 raft_consensus.cc:2781] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:29:17.045118 17245 raft_consensus.cc:482] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:29:17.045212 17245 raft_consensus.cc:3037] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:29:17.046258 17245 raft_consensus.cc:504] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.046622 17173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b1dad2a7124b08914733cb2726ae58" candidate_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093956dee3f443e48ea11ef51d367f2f"
I20260924 15:29:17.046708 17173 raft_consensus.cc:3037] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:29:17.047642 17173 raft_consensus.cc:2454] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f8a18d51b54d15b13830b80ea0ba08 in term 1.
I20260924 15:29:17.047907 16973 leader_election.cc:302] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093956dee3f443e48ea11ef51d367f2f, e4f8a18d51b54d15b13830b80ea0ba08; no voters:
I20260924 15:29:17.048187 16901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52b1dad2a7124b08914733cb2726ae58" candidate_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86689e0a3474ef6bff98257a901f05e"
I20260924 15:29:17.048280 16901 raft_consensus.cc:3037] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:29:17.049191 16901 raft_consensus.cc:2454] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f8a18d51b54d15b13830b80ea0ba08 in term 1.
I20260924 15:29:17.049432 17245 leader_election.cc:288] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [CANDIDATE]: Term 1 election: Requested vote from peers 093956dee3f443e48ea11ef51d367f2f (127.5.98.195:40611), d86689e0a3474ef6bff98257a901f05e (127.5.98.193:33317)
I20260924 15:29:17.049535 17245 raft_consensus.cc:2781] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:29:17.049611 17245 raft_consensus.cc:686] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 1 LEADER]: Becoming Leader. State: Replica: e4f8a18d51b54d15b13830b80ea0ba08, State: Running, Role: LEADER
I20260924 15:29:17.049738 17245 consensus_queue.cc:237] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } }
I20260924 15:29:17.050712 16733 catalog_manager.cc:5495] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4f8a18d51b54d15b13830b80ea0ba08 (127.5.98.194). New cstate: current_term: 1 leader_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f8a18d51b54d15b13830b80ea0ba08" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 42439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 } health_report { overall_health: UNKNOWN } } }
I20260924 15:29:17.062346 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.058330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.5.98.194:42439)
0924 15:29:17.058405 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:29:17.058407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:29:17.058416 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:29:17.058455 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:29:17.058456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:29:17.058483 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:29:17.058561 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:29:17.058776 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.059553 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.061087 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:29:17.061409 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:29:17.061485 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:29:17.061706 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:29:17.061860 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:29:17.061953 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:29:17.062217 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:29:17.062241 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 15:29:17.076508 16901 raft_consensus.cc:1261] T 52b1dad2a7124b08914733cb2726ae58 P d86689e0a3474ef6bff98257a901f05e [term 1 FOLLOWER]: Refusing update from remote peer e4f8a18d51b54d15b13830b80ea0ba08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:29:17.077023 17247 consensus_queue.cc:1035] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d86689e0a3474ef6bff98257a901f05e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:29:17.079619 17173 raft_consensus.cc:1261] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 1 FOLLOWER]: Refusing update from remote peer e4f8a18d51b54d15b13830b80ea0ba08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:29:17.080034 17247 consensus_queue.cc:1035] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093956dee3f443e48ea11ef51d367f2f" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:29:17.085832 17250 mvcc.cc:205] Tried to move back new op lower bound from 7332920348973191168 to 7332920348876734464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920348973191168}]
I20260924 15:29:17.085996 17251 mvcc.cc:205] Tried to move back new op lower bound from 7332920348973191168 to 7332920348876734464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920348973191168}]
I20260924 15:29:17.092199 16733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:29:17.092289 16733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:29:17.093123 16733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540332d4bea248f7b918c31fcd808292: Sending DeleteTablet for 3 replicas of tablet 52b1dad2a7124b08914733cb2726ae58
Sep 24 15:29:17 dist-test-slave-4g6f krb5kdc[16257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:17.094265 17153 tablet_service.cc:1514] Processing DeleteTablet for tablet 52b1dad2a7124b08914733cb2726ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:29:17 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:55070
I20260924 15:29:17.094266 16881 tablet_service.cc:1514] Processing DeleteTablet for tablet 52b1dad2a7124b08914733cb2726ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:29:17 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:56480
I20260924 15:29:17.094664 17019 tablet_service.cc:1514] Processing DeleteTablet for tablet 52b1dad2a7124b08914733cb2726ae58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:29:17 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:44178
I20260924 15:29:17.095227 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 16815
I20260924 15:29:17.099421 17253 mvcc.cc:205] Tried to move back new op lower bound from 7332920348973191168 to 7332920348876734464. Current Snapshot: MvccSnapshot[applied={T|T < 7332920348973191168}]
I20260924 15:29:17.101881 17258 tablet_replica.cc:330] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: stopping tablet replica
I20260924 15:29:17.102072 17258 raft_consensus.cc:2229] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:29:17.102226 17258 raft_consensus.cc:2258] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:29:17.106623 17260 tablet_replica.cc:330] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: stopping tablet replica
I20260924 15:29:17.106809 17260 raft_consensus.cc:2229] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:29:17.106995 17260 raft_consensus.cc:2258] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:29:17.107578 17260 ts_tablet_manager.cc:1905] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:29:17.109578 17260 ts_tablet_manager.cc:1918] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:29:17.109709 17260 log.cc:1196] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/52b1dad2a7124b08914733cb2726ae58
I20260924 15:29:17.110077 17260 ts_tablet_manager.cc:1939] T 52b1dad2a7124b08914733cb2726ae58 P e4f8a18d51b54d15b13830b80ea0ba08: Deleting consensus metadata
I20260924 15:29:17.110589 16720 catalog_manager.cc:4841] TS e4f8a18d51b54d15b13830b80ea0ba08 (127.5.98.194:42439): tablet 52b1dad2a7124b08914733cb2726ae58 (table test-table [id=28ab0a2b6f7b4de2953502cba2c1ad86]) successfully deleted
I20260924 15:29:17.112313 17258 ts_tablet_manager.cc:1905] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:29:17.113867 17258 ts_tablet_manager.cc:1918] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:29:17.118439 17258 log.cc:1196] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/52b1dad2a7124b08914733cb2726ae58
I20260924 15:29:17.119179 17258 ts_tablet_manager.cc:1939] T 52b1dad2a7124b08914733cb2726ae58 P 093956dee3f443e48ea11ef51d367f2f: Deleting consensus metadata
W20260924 15:29:17.119272 16718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:33317 (error 108)
I20260924 15:29:17.119756 16718 catalog_manager.cc:4841] TS 093956dee3f443e48ea11ef51d367f2f (127.5.98.195:40611): tablet 52b1dad2a7124b08914733cb2726ae58 (table test-table [id=28ab0a2b6f7b4de2953502cba2c1ad86]) successfully deleted
I20260924 15:29:17.119907 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 16955
I20260924 15:29:17.122885 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:29:17.122775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33317 (local address 127.0.0.1:56484)
0924 15:29:17.122821 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:29:17.122852 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:29:17.123008 16718 catalog_manager.cc:4584] TS d86689e0a3474ef6bff98257a901f05e (127.5.98.193:33317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52b1dad2a7124b08914733cb2726ae58: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33317: connect: Connection refused (error 111)
I20260924 15:29:17.127488 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 17090
I20260924 15:29:17.134311 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 16701
I20260924 15:29:17.140677 5515 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:29:17.496261 5515 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:29:17.496366 5515 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:29:17.840889 5515 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:29:17.841 UTC [16287] LOG: received smart shutdown request
2026-09-24 15:29:17.844 UTC [16287] LOG: background worker "logical replication launcher" (PID 16296) exited with exit code 1
2026-09-24 15:29:17.844 UTC [16291] LOG: shutting down
2026-09-24 15:29:18.001 UTC [16287] LOG: database system is shut down
2026-09-24T15:29:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48408 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:29:18 dist-test-slave-4g6f krb5kdc[17271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:29:18 dist-test-slave-4g6f krb5kdc[17271](info): set up 2 sockets
Sep 24 15:29:18 dist-test-slave-4g6f krb5kdc[17271](info): commencing operation
krb5kdc: starting...
W20260924 15:29:19.157441 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:29:19 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:29:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangeradmin_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangeradmin_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangerlookup_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangerlookup_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
I20260924 15:29:19.324331 5515 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskC7bDtI/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/postgres -l logfile start
2026-09-24 15:29:21.804 UTC [17301] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:29:21.804 UTC [17301] LOG: listening on IPv4 address "127.5.98.212", port 57297
2026-09-24 15:29:21.808 UTC [17301] LOG: listening on Unix socket "/tmp/.s.PGSQL.57297"
2026-09-24 15:29:21.829 UTC [17304] LOG: database system was shut down at 2026-09-24 15:29:20 UTC
2026-09-24 15:29:21.833 UTC [17301] LOG: database system is ready to accept connections
I20260924 15:29:22.792635 5515 mini_postgres.cc:96] Postgres bound to 57297
2026-09-24 15:29:22.798 UTC [17312] FATAL: database "slave" does not exist
127.5.98.212:57297 - accepting connections
I20260924 15:29:22.799201 5515 mini_ranger.cc:161] Starting Ranger...
I20260924 15:29:22.813923 5515 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:29:23.294188 5515 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:29:23.294260 5515 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin
2026-09-24 15:29:23,566 [I] DB FLAVOR :POSTGRES
2026-09-24 15:29:23,567 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:29:23,567 [I] Checking connection..
2026-09-24 15:29:23,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:29:23,828 [I] Checking connection passed.
2026-09-24 15:29:23,828 [I] --------- Verifying version history table ---------
2026-09-24 15:29:23,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:29:24,064 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:29:24,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:29:24,309 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:29:24,309 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:29:24,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:29:24.530 UTC [17376] WARNING: there is no transaction in progress
2026-09-24 15:29:24,536 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:29:24,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:29:24,785 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:29:24,785 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:29:24,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:29:25,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:29:25,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4g6f.c.gcp-upstream.internal','N') ;"
2026-09-24 15:29:25,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:29:25,761 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:29:25,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:29:26,006 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:29:26,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:29:26,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:29:26,480 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:29:26,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:29:27.401 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.417 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.432 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.447 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.464 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.480 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.534 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.540 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.554 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.564 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.574 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.588 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.597 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.603 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.610 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:27.616 UTC [17484] WARNING: there is no transaction in progress
2026-09-24 15:29:28,243 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:29:28,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:29:28,484 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:29:28,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:29:28,741 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:29:28,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:29:28,999 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:29:28,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:29:29,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:29:29,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4g6f.c.gcp-upstream.internal';"
2026-09-24 15:29:29,725 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:29:29,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:29:29,957 [I] DB_PATCHES have already been applied
I20260924 15:29:29.962533 5515 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskC7bDtI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:29:29.962597 5515 mini_ranger.cc:193] Using host: 127.5.98.212
I20260924 15:29:29.965734 5515 mini_ranger.cc:227] Ranger admin URL: http://127.5.98.212:34771
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:29:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:29:31.081902 5515 mini_ranger.cc:160] Time spent starting Ranger: real 8.283s user 0.000s sys 0.008s
Sep 24, 2026 3:29:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:29:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 3:29:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:29:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:29:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:29:52.316406 5515 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.98.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.98.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangerkms_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.98.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/rangerkms_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.98.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.98.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.98.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/HTTP_127.5.98.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:29:52 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263792, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:29:52.400738 5515 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:29:52.410072 5515 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:29:53.287173 5515 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:29:53.287292 5515 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms
I20260924 15:29:53.287364 5515 mini_ranger_kms.cc:227] Using postgres at 127.5.98.212:57297
2026-09-24 15:29:53,467 [I] DB FLAVOR :POSTGRES
2026-09-24 15:29:53,467 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:29:53,467 [I] Checking connection
2026-09-24 15:29:53,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:29:53,731 [I] connection success
2026-09-24 15:29:53,731 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:29:53,731 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:29:53,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:29:53,966 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:29:53,966 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:29:53,966 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:29:53,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.98.212:57297/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:29:54,227 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:29:54.728801 5515 mini_ranger_kms.cc:310] Created kms service
I20260924 15:29:54.939323 5515 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:29:54.984903 5515 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:29:54 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263792, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:29:55.362802 5515 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:29:55.362938 5515 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms:/tmp/dist-test-taskC7bDtI/build/release/bin/postgresql.jar:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskC7bDtI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:29:55.362962 5515 mini_ranger_kms.cc:240] Using host: 127.5.98.212
I20260924 15:29:55.367225 5515 mini_ranger_kms.cc:276] Ranger KMS PID: 17664
I20260924 15:29:55.367309 5515 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.98.212:39371
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:29:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 15:29:57.614256 5515 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.214s user 0.014s sys 0.149s
I20260924 15:29:57.614363 5515 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:29:57.614409 5515 mini_ranger_kms.cc:401] 127.5.98.212:39371/kms/v1/keys
Sep 24 15:29:57 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 15:29:58 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
Sep 24 15:29:58 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.98.212@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
Sep 24 15:30:00 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:30:00.698624 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33037
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:33037
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:00.792246 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:00.792465 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:00.792498 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:00.793593 17715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:00.793665 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:00.793690 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:00.793717 17715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:00.793754 17715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:00.795451 17715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.98.212:39371/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:33037
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:00.795738 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:00.795964 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:00.802290 17721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:00.802353 17715 server_base.cc:1034] running on GCE node
W20260924 15:30:00.802451 17720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:00.802302 17723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:00.802860 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:00.803228 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:00.804504 17715 hybrid_clock.cc:648] HybridClock initialized: now 1790263800804488 us; error 29 us; skew 500 ppm
Sep 24 15:30:00 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263800, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:00.811492 17715 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:00.812003 17715 webserver.cc:460] Webserver started at http://127.5.98.254:39121/ using document root <none> and password file <none>
I20260924 15:30:00.812248 17715 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:00.812309 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:00.812464 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:30:00 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263800, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:00.865329 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "f33c950a34e94bbb9e09918332950222"
format_stamp: "Formatted at 2026-09-24 15:30:00 on dist-test-slave-4g6f"
server_key: "99074dc0e1c4802ba8cfe3a6602bc8d0"
server_key_iv: "3218ddf8ee77a5f235d8a23b4d6a1203"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:00.865726 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "f33c950a34e94bbb9e09918332950222"
format_stamp: "Formatted at 2026-09-24 15:30:00 on dist-test-slave-4g6f"
server_key: "99074dc0e1c4802ba8cfe3a6602bc8d0"
server_key_iv: "3218ddf8ee77a5f235d8a23b4d6a1203"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:00.867192 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:00.882063 17730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:00.882289 17715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:00.882381 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "f33c950a34e94bbb9e09918332950222"
format_stamp: "Formatted at 2026-09-24 15:30:00 on dist-test-slave-4g6f"
server_key: "99074dc0e1c4802ba8cfe3a6602bc8d0"
server_key_iv: "3218ddf8ee77a5f235d8a23b4d6a1203"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:00.882477 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:00.897686 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:00.899825 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:00.899993 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:00.904858 17715 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:33037
I20260924 15:30:00.904937 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:33037 every 8 connection(s)
I20260924 15:30:00.905337 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:00.906028 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:00.908341 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Bootstrap starting.
I20260924 15:30:00.908911 17784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:00.909198 17784 log.cc:824] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:00.909843 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: No bootstrap required, opened a new log
I20260924 15:30:00.911806 17784 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:00.911947 17784 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:00.911968 17784 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33c950a34e94bbb9e09918332950222, State: Initialized, Role: FOLLOWER
I20260924 15:30:00.912132 17784 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:00.912211 17784 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:00.912261 17784 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:00.912307 17784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:00.912853 17784 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:00.912966 17784 leader_election.cc:302] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33c950a34e94bbb9e09918332950222; no voters:
I20260924 15:30:00.913167 17784 leader_election.cc:288] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:30:00.913189 17787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:00.913314 17787 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 LEADER]: Becoming Leader. State: Replica: f33c950a34e94bbb9e09918332950222, State: Running, Role: LEADER
I20260924 15:30:00.913429 17784 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:00.913460 17787 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:00.913846 17787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33c950a34e94bbb9e09918332950222. Latest consensus state: current_term: 1 leader_uuid: "f33c950a34e94bbb9e09918332950222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } } }
I20260924 15:30:00.913856 17789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f33c950a34e94bbb9e09918332950222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } } }
I20260924 15:30:00.914050 17787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:00.914135 17789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:00.914364 17796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:00.914861 17796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:00.915146 17784 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:30:00.916569 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 17715
I20260924 15:30:00.916673 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:30:00.916863 5515 external_mini_cluster.cc:1428] Setting key b32d67eacbeeaa0182e5c98c4a01e2fa
Sep 24 15:30:00 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:00.931644 17796 catalog_manager.cc:1347] Generated new cluster ID: 69c8831f635f44faabcab7299a12da5d
I20260924 15:30:00.931720 17796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:00.944530 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:00.918119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.5.98.254:33037)
0924 15:30:00.920536 (+ 2417us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:00.920546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:00.920578 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:30:00.920768 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:00.920776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:00.920853 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:00.920975 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:00.921978 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:00.923558 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:00.924632 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:00.925258 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:00.925380 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:00.928387 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:00.928391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:00.928392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:00.928404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:00.930323 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:00.930702 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:00.930705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:00.930705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:00.930797 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:00.930981 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:00.930982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:00.930982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:00.931296 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:00.931448 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:00.944148 (+ 12700us) server_negotiation.cc:299] Negotiation successful
0924 15:30:00.944289 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":243,"threads_started":1}
I20260924 15:30:00.971186 17796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:30:00.972321 17796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:00.997671 17796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Generated new TSK 0
I20260924 15:30:00.998036 17796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:30:01.706887 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:01.809396 17829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:01.809628 17829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:01.809669 17829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:01.810847 17829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:01.810913 17829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:01.810957 17829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:01.812450 17829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:01.812707 17829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:01.812925 17829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:01.818287 17834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:01.818562 17835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:01.818346 17837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:01.818819 17829 server_base.cc:1034] running on GCE node
I20260924 15:30:01.819167 17829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:01.819422 17829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:01.820597 17829 hybrid_clock.cc:648] HybridClock initialized: now 1790263801820555 us; error 56 us; skew 500 ppm
Sep 24 15:30:01 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263801, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:01.824136 17829 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:01.824620 17829 webserver.cc:460] Webserver started at http://127.5.98.193:34887/ using document root <none> and password file <none>
I20260924 15:30:01.824855 17829 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:01.824927 17829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:01.825049 17829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:30:01 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263801, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:01.839854 17829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "78a30e6068b44b6bafb32f82f65f7cef"
format_stamp: "Formatted at 2026-09-24 15:30:01 on dist-test-slave-4g6f"
server_key: "83a055d5c5bc084803f348c28d521541"
server_key_iv: "505d5fa0d8d827ce7eb1af9362f49278"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:01.840337 17829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "78a30e6068b44b6bafb32f82f65f7cef"
format_stamp: "Formatted at 2026-09-24 15:30:01 on dist-test-slave-4g6f"
server_key: "83a055d5c5bc084803f348c28d521541"
server_key_iv: "505d5fa0d8d827ce7eb1af9362f49278"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:01.842101 17829 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:01.857983 17844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:01.858170 17829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:01.858263 17829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "78a30e6068b44b6bafb32f82f65f7cef"
format_stamp: "Formatted at 2026-09-24 15:30:01 on dist-test-slave-4g6f"
server_key: "83a055d5c5bc084803f348c28d521541"
server_key_iv: "505d5fa0d8d827ce7eb1af9362f49278"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:01.858368 17829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:01.890604 17829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:01.892643 17829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:01.892817 17829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:01.893074 17829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:01.893473 17829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:01.893527 17829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:01.893564 17829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:01.893589 17829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:01.900830 17829 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:44609
I20260924 15:30:01.901189 17829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:30:01.901801 17960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:44609 every 8 connection(s)
I20260924 15:30:01.905995 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 17829
I20260924 15:30:01.906093 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:30:01.906288 5515 external_mini_cluster.cc:1428] Setting key a98a7fffef96226229d962e8a7783f6b
Sep 24 15:30:01 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263801, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:01.911693 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:01.901970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.193:56931)
0924 15:30:01.902188 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:01.902252 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:01.902772 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:01.903055 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:01.903060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:01.903245 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:01.903434 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:01.903438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:01.904237 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:01.904239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:01.905367 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:01.905370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:01.905981 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:01.905982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:01.906121 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:01.906699 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:01.906709 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:01.908221 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:01.910184 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:01.910190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:01.910192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:01.910408 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:01.910627 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:01.910628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:01.910628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:01.910742 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:01.911029 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:01.911035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:01.911190 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 15:30:01.911294 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:30:01.911439 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260924 15:30:01.912577 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:01.901960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:56931 (local address 127.5.98.254:33037)
0924 15:30:01.902121 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:01.902124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:01.902347 (+ 223us) server_negotiation.cc:407] Connection header received
0924 15:30:01.902882 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:01.902884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:01.902898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:01.902964 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:01.903553 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:01.904139 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:01.905465 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:01.905829 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:01.905893 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:01.908319 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:01.908320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:01.908321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:01.908328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:01.910090 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:01.910488 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:01.910489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:01.910489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:01.910550 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:01.910819 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:01.910820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:01.910821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:01.910948 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:01.912339 (+ 1391us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:01.912392 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 15:30:01.912488 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260924 15:30:01.913107 17961 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:01.913196 17961 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:01.913378 17961 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:01.913890 17748 ts_manager.cc:194] Registered new tserver with Master: 78a30e6068b44b6bafb32f82f65f7cef (127.5.98.193:44609)
I20260924 15:30:01.914491 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:56931
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:30:01.938092 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:02.056375 17967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:02.056634 17967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:02.056669 17967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:02.058295 17967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:02.058357 17967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:02.058398 17967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:02.060511 17967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:02.060752 17967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:02.061036 17967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 15:30:02.067546 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.067752 17967 server_base.cc:1034] running on GCE node
W20260924 15:30:02.067936 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.067538 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.068187 17967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:02.068466 17967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:02.069648 17967 hybrid_clock.cc:648] HybridClock initialized: now 1790263802069619 us; error 39 us; skew 500 ppm
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:02.072660 17967 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:02.073184 17967 webserver.cc:460] Webserver started at http://127.5.98.194:40983/ using document root <none> and password file <none>
I20260924 15:30:02.073483 17967 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:02.073551 17967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:02.073685 17967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:02.098186 17967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "9d151362610d43dbbc8c6369259af19c"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "e8ce5a30cfd5b434850398afc0127589"
server_key_iv: "d81fd6c57b0493f3aa9ba3098ee069f2"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.098690 17967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "9d151362610d43dbbc8c6369259af19c"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "e8ce5a30cfd5b434850398afc0127589"
server_key_iv: "d81fd6c57b0493f3aa9ba3098ee069f2"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.100665 17967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:02.127521 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.127763 17967 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 15:30:02.127894 17967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "9d151362610d43dbbc8c6369259af19c"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "e8ce5a30cfd5b434850398afc0127589"
server_key_iv: "d81fd6c57b0493f3aa9ba3098ee069f2"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.128008 17967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:02.135560 17967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:02.137693 17967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:02.137871 17967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:02.138141 17967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:02.138545 17967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:02.138602 17967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.138639 17967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:02.138666 17967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.146047 17967 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:39681
I20260924 15:30:02.146435 17967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:30:02.147531 18098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:39681 every 8 connection(s)
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:02.154843 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 17967
I20260924 15:30:02.154942 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:30:02.155180 5515 external_mini_cluster.cc:1428] Setting key c2e4701ae5ff9e1eaf29b285ea385fa3
I20260924 15:30:02.157121 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.147410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:36903 (local address 127.5.98.254:33037)
0924 15:30:02.147460 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:02.147462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:02.148075 (+ 613us) server_negotiation.cc:407] Connection header received
0924 15:30:02.148638 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:02.148639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:02.148656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:02.148707 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:02.149273 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.149788 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.150899 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.151120 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.151200 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.153321 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:02.153322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:02.153323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:02.153337 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:02.155041 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.155880 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.155881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.155882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.155944 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.156258 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.156259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.156259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.156408 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:02.156462 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:02.156836 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 15:30:02.156937 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:30:02.157770 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.147528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.194:36903)
0924 15:30:02.147937 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:02.147976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:02.148522 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:02.148777 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:02.148781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:02.148979 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:02.149172 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.149176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.149909 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.149910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.150790 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.150793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.151220 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.151221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.151345 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.151963 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:02.151971 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:02.153216 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:02.155156 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.155162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.155165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.155778 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.156023 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.156024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.156024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.156180 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.156497 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:02.156503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:02.156674 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 15:30:02.157472 (+ 798us) client_negotiation.cc:240] Negotiation successful
0924 15:30:02.157609 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":92,"threads_started":1}
I20260924 15:30:02.158236 18099 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:02.158402 18099 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:02.158645 18099 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:02.159057 17748 ts_manager.cc:194] Registered new tserver with Master: 9d151362610d43dbbc8c6369259af19c (127.5.98.194:39681)
I20260924 15:30:02.159587 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:36903
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:30:02.182425 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:02.303910 18105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:02.304196 18105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:02.304332 18105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:02.306111 18105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:02.306172 18105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:02.306214 18105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:02.308429 18105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:02.308898 18105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:02.309239 18105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:02.316058 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.316236 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.316376 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.316633 18105 server_base.cc:1034] running on GCE node
I20260924 15:30:02.316865 18105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:02.317166 18105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:02.318398 18105 hybrid_clock.cc:648] HybridClock initialized: now 1790263802318349 us; error 61 us; skew 500 ppm
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:02.321852 18105 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:02.322379 18105 webserver.cc:460] Webserver started at http://127.5.98.195:44489/ using document root <none> and password file <none>
I20260924 15:30:02.322688 18105 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:02.322763 18105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:02.322956 18105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:02.339417 18105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "a6838f418cc745ba9684c053d3cae535"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "a8d30a201ca9cb8903a3166a27d8e84f"
server_key_iv: "989d37989bb182b47042d52aca591a23"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.339924 18105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "a6838f418cc745ba9684c053d3cae535"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "a8d30a201ca9cb8903a3166a27d8e84f"
server_key_iv: "989d37989bb182b47042d52aca591a23"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.341828 18105 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:02.362481 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.362697 18105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260924 15:30:02.362828 18105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "a6838f418cc745ba9684c053d3cae535"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "a8d30a201ca9cb8903a3166a27d8e84f"
server_key_iv: "989d37989bb182b47042d52aca591a23"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.362944 18105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:02.374907 18105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:02.377136 18105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:02.377259 18105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:02.377475 18105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:02.377770 18105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:02.377801 18105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.377828 18105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:02.377844 18105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.386197 18105 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:36005
I20260924 15:30:02.386708 18105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:02.387405 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:36005 every 8 connection(s)
I20260924 15:30:02.388500 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18105
I20260924 15:30:02.388602 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:30:02.388820 5515 external_mini_cluster.cc:1428] Setting key 82f9200a3683e1a329893c400df2c265
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:02.399783 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.388194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:41985 (local address 127.5.98.254:33037)
0924 15:30:02.388242 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:02.388245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:02.389449 (+ 1204us) server_negotiation.cc:407] Connection header received
0924 15:30:02.390329 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:02.390330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:02.390342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:02.390390 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:02.390994 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.391581 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.393005 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.393263 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.393302 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.396014 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:02.396016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:02.396016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:02.396024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:02.398052 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.398583 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.398585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.398585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.398663 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.398955 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.398956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.398957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.399097 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:02.399141 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:02.399549 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 15:30:02.399675 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:30:02.401328 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.388154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.195:41985)
0924 15:30:02.389133 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:02.389254 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:02.390202 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:02.390468 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:02.390472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:02.390685 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:02.390895 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.390899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.391699 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.391700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.392893 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.392896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.393403 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.393404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.393510 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.394103 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:02.394112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:02.395891 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:02.398155 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.398162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.398165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.398489 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.398740 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.398741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.398741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.398873 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.399189 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:02.399194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:02.399374 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 15:30:02.400959 (+ 1585us) client_negotiation.cc:240] Negotiation successful
0924 15:30:02.401148 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":98,"threads_started":1}
I20260924 15:30:02.401883 18235 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:02.402001 18235 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:02.402199 18235 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:02.402789 17748 ts_manager.cc:194] Registered new tserver with Master: a6838f418cc745ba9684c053d3cae535 (127.5.98.195:36005)
I20260924 15:30:02.403218 17748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:41985
I20260924 15:30:02.411643 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:02.412387 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 17829
I20260924 15:30:02.420475 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 17967
I20260924 15:30:02.425951 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18105
I20260924 15:30:02.431991 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 17715
I20260924 15:30:02.439182 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33037
--webserver_interface=127.5.98.254
--webserver_port=39121
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:33037
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:02.521046 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:02.521237 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:02.521258 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:02.522374 18238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:02.522415 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:02.522430 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:02.522466 18238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:02.522495 18238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:02.524070 18238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:33037
--ranger_config_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:02.524308 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:02.524533 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:02.529295 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.529349 18238 server_base.cc:1034] running on GCE node
W20260924 15:30:02.529320 18244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.529358 18245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.529731 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:02.530036 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:02.531199 18238 hybrid_clock.cc:648] HybridClock initialized: now 1790263802531196 us; error 38 us; skew 500 ppm
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:02.533738 18238 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:02.534150 18238 webserver.cc:460] Webserver started at http://127.5.98.254:39121/ using document root <none> and password file <none>
I20260924 15:30:02.534368 18238 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:02.534442 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:02.581378 18238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:02.582263 18256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.582448 18238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:02.582559 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "f33c950a34e94bbb9e09918332950222"
format_stamp: "Formatted at 2026-09-24 15:30:00 on dist-test-slave-4g6f"
server_key: "99074dc0e1c4802ba8cfe3a6602bc8d0"
server_key_iv: "3218ddf8ee77a5f235d8a23b4d6a1203"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.583036 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:02.599097 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:02.600876 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:02.600998 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:02.605067 18238 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:33037
I20260924 15:30:02.605113 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:33037 every 8 connection(s)
I20260924 15:30:02.605479 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:02.606714 18309 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:30:02.607466 18309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Bootstrap starting.
I20260924 15:30:02.614871 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18238
I20260924 15:30:02.615217 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:44609
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:33037
--webserver_port=34887
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:02.615569 18309 log.cc:824] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:02.617468 18309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:02.617780 18309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Bootstrap complete.
I20260924 15:30:02.622380 18309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:02.622826 18309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33c950a34e94bbb9e09918332950222, State: Initialized, Role: FOLLOWER
I20260924 15:30:02.623027 18309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:02.623124 18309 raft_consensus.cc:388] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:02.623190 18309 raft_consensus.cc:482] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:02.623251 18309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:30:02.625049 18309 raft_consensus.cc:504] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:02.625190 18309 leader_election.cc:302] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f33c950a34e94bbb9e09918332950222; no voters:
I20260924 15:30:02.625398 18309 leader_election.cc:288] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:30:02.625468 18313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:30:02.625648 18313 raft_consensus.cc:686] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [term 2 LEADER]: Becoming Leader. State: Replica: f33c950a34e94bbb9e09918332950222, State: Running, Role: LEADER
I20260924 15:30:02.625706 18309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:02.625773 18313 consensus_queue.cc:237] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } }
I20260924 15:30:02.626021 18313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f33c950a34e94bbb9e09918332950222. Latest consensus state: current_term: 2 leader_uuid: "f33c950a34e94bbb9e09918332950222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } } }
I20260924 15:30:02.626132 18313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:02.626022 18316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f33c950a34e94bbb9e09918332950222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f33c950a34e94bbb9e09918332950222" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 33037 } } }
I20260924 15:30:02.626230 18316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:02.626376 18318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:02.626884 18318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:02.627035 18318 catalog_manager.cc:1259] Loaded cluster ID: 69c8831f635f44faabcab7299a12da5d
I20260924 15:30:02.627079 18318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:02.627468 18309 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:30:02.631131 18318 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:02.631804 18318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Loaded TSK: 0
I20260924 15:30:02.632128 18318 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:30:02.716671 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:02.716907 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:02.716951 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:02.718439 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:02.718516 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:02.718567 18312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:02.720270 18312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:44609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=34887
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:02.720574 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:02.720816 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:02.726807 18350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.727167 18312 server_base.cc:1034] running on GCE node
W20260924 15:30:02.726874 18348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.727372 18347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.727639 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:02.728004 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:02.729171 18312 hybrid_clock.cc:648] HybridClock initialized: now 1790263802729160 us; error 30 us; skew 500 ppm
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:02.732388 18312 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:02.732909 18312 webserver.cc:460] Webserver started at http://127.5.98.193:34887/ using document root <none> and password file <none>
I20260924 15:30:02.733186 18312 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:02.733281 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:02.755311 18312 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 15:30:02.756381 18358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.756572 18312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:02.756672 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "78a30e6068b44b6bafb32f82f65f7cef"
format_stamp: "Formatted at 2026-09-24 15:30:01 on dist-test-slave-4g6f"
server_key: "83a055d5c5bc084803f348c28d521541"
server_key_iv: "505d5fa0d8d827ce7eb1af9362f49278"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.757091 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:02.768487 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:02.770774 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:02.770949 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:02.771210 18312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:02.771620 18312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:02.771713 18312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.771756 18312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:02.771781 18312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.779683 18312 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:44609
I20260924 15:30:02.780232 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:30:02.781762 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18312
I20260924 15:30:02.782054 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:39681
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:33037
--webserver_port=40983
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:02.785609 18473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:44609 every 8 connection(s)
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:02.809816 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.786367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.193:39615)
0924 15:30:02.791764 (+ 5397us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:02.791834 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:02.792561 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:02.793321 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:02.793326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:02.793612 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:02.793968 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.793976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.796157 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.796159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.797559 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.797561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.798572 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.798573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.798942 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.799570 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:02.799580 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:02.804640 (+ 5060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:02.807774 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.807781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.807783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.808114 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.808396 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.808396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.808397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.808522 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.809116 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:02.809122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:02.809282 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 15:30:02.809392 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:30:02.809559 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5339,"thread_start_us":93,"threads_started":1}
I20260924 15:30:02.810544 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.786785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:39615 (local address 127.5.98.254:33037)
0924 15:30:02.791859 (+ 5074us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:02.791870 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:02.792120 (+ 250us) server_negotiation.cc:407] Connection header received
0924 15:30:02.792850 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:02.792859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:02.792929 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:02.793112 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:02.794126 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.795953 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.797666 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.798353 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.798457 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.804806 (+ 6349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:02.804812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:02.804812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:02.804828 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:02.807617 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.808207 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.808211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.808211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.808321 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.808592 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.808593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.808594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.809002 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:02.809086 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:02.810159 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0924 15:30:02.810287 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4987,"thread_start_us":102,"threads_started":1}
I20260924 15:30:02.811228 18474 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:02.811336 18474 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:02.811554 18474 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:02.812426 18474 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:30:02.812506 18474 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:02.812577 18474 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:02.812896 18474 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:02.812975 18474 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:02.813272 18474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:30:02.904770 18475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:02.905144 18475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:02.905234 18475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:02.907122 18475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:02.907248 18475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:02.907452 18475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:02.909909 18475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:39681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=40983
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:02.910439 18475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:02.910818 18475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:02.917564 18484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.917735 18486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:02.917829 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:02.918013 18475 server_base.cc:1034] running on GCE node
I20260924 15:30:02.918286 18475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:02.918581 18475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:02.919745 18475 hybrid_clock.cc:648] HybridClock initialized: now 1790263802919712 us; error 50 us; skew 500 ppm
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:02.923229 18475 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:02.923885 18475 webserver.cc:460] Webserver started at http://127.5.98.194:40983/ using document root <none> and password file <none>
I20260924 15:30:02.924196 18475 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:02.924288 18475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:02.944238 18475 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260924 15:30:02.946668 18494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.946857 18475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 15:30:02.946956 18475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "9d151362610d43dbbc8c6369259af19c"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "e8ce5a30cfd5b434850398afc0127589"
server_key_iv: "d81fd6c57b0493f3aa9ba3098ee069f2"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:02.947492 18475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:02.972321 18475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:02.974881 18475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:02.975078 18475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:02.975391 18475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:02.975857 18475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:02.975914 18475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.975948 18475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:02.975978 18475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:02.984331 18475 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:39681
I20260924 15:30:02.984428 18609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:39681 every 8 connection(s)
I20260924 15:30:02.984886 18475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:30:02 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263802, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:02.992409 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18475
I20260924 15:30:02.992677 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.98.212:39371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:36005
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:33037
--webserver_port=44489
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:02.998198 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.985744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.194:55579)
0924 15:30:02.985916 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:02.985959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:02.986525 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:02.986858 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:02.986862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:02.987041 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:02.987278 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.987284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.988725 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.988726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.989705 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:02.989707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.990139 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.990141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:02.990265 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.990827 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:02.990837 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:02.994169 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:02.996326 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.996334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.996337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.996654 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.996885 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:02.996886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:02.996886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:02.997035 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:02.997367 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:02.997374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:02.997560 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 15:30:02.997683 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 15:30:02.997884 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260924 15:30:02.999085 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:02.985737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:55579 (local address 127.5.98.254:33037)
0924 15:30:02.985795 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:02.985797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:02.986080 (+ 283us) server_negotiation.cc:407] Connection header received
0924 15:30:02.986647 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:02.986648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:02.986665 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:02.986751 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:02.987417 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.988250 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.989797 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:02.990019 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:02.990107 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:02.994285 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:02.994287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:02.994287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:02.994297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:02.996232 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.996739 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.996740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.996741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.996800 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:02.997117 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:02.997118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:02.997119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:02.997272 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:02.997338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:02.998854 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0924 15:30:02.998957 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:30:02.999704 18610 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:02.999874 18610 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.000141 18610 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:03.000820 18610 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:30:03.000968 18610 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.001083 18610 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:03.001427 18610 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.001549 18610 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:03.001893 18610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:30:03.080727 18613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:03.080957 18613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:03.080996 18613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:03.082265 18613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:03.082326 18613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:03.082376 18613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:03.083973 18613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.98.212:39371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:36005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=44489
--tserver_master_addrs=127.5.98.254:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:03.084218 18613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:03.084451 18613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:03.089582 18621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:03.089599 18619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:03.089978 18618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:03.090281 18613 server_base.cc:1034] running on GCE node
I20260924 15:30:03.090507 18613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:03.090736 18613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:03.091910 18613 hybrid_clock.cc:648] HybridClock initialized: now 1790263803091894 us; error 34 us; skew 500 ppm
Sep 24 15:30:03 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263803, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:03.095077 18613 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:03.095569 18613 webserver.cc:460] Webserver started at http://127.5.98.195:44489/ using document root <none> and password file <none>
I20260924 15:30:03.095846 18613 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:03.095911 18613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:30:03 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263803, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:03.123282 18613 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:03.124342 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:03.124511 18613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:03.124588 18613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "a6838f418cc745ba9684c053d3cae535"
format_stamp: "Formatted at 2026-09-24 15:30:02 on dist-test-slave-4g6f"
server_key: "a8d30a201ca9cb8903a3166a27d8e84f"
server_key_iv: "989d37989bb182b47042d52aca591a23"
server_key_version: "kuduclusterkey@0"
I20260924 15:30:03.125056 18613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:03.138051 18613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:03.140436 18613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:03.140594 18613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:03.140811 18613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:03.141160 18613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:03.141204 18613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:03.141240 18613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:03.141266 18613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:03.152066 18613 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:36005
I20260924 15:30:03.152570 18613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:03.153187 18744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:36005 every 8 connection(s)
Sep 24 15:30:03 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263803, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:03.159421 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18613
I20260924 15:30:03.162746 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:03.153521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:33037 (local address 127.5.98.195:44447)
0924 15:30:03.153705 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:03.153742 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:03.154236 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:03.154528 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:03.154532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:03.154700 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:03.154900 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:03.154904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.155653 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.155699 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:03.156513 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:03.156515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.156953 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.156955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:03.157070 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:03.157615 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:03.157624 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:03.158862 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:03.161132 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:03.161138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:03.161141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:03.161444 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:03.161643 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:03.161644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:03.161644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:03.161770 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:03.162044 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:03.162050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:03.162251 (+ 201us) client_negotiation.cc:769] Sending connection context
0924 15:30:03.162361 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:30:03.162512 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260924 15:30:03.163093 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:03.153492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:44447 (local address 127.5.98.254:33037)
0924 15:30:03.153540 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:03.153543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:03.153842 (+ 299us) server_negotiation.cc:407] Connection header received
0924 15:30:03.154347 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:03.154349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:03.154363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:03.154429 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:03.154999 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.155557 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.156621 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.156836 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.156922 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:03.158963 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:03.158965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:03.158965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:03.158973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:03.161037 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:03.161518 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:03.161519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:03.161519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:03.161577 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:03.161850 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:03.161851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:03.161852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:03.161967 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:03.162021 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:03.162922 (+ 901us) server_negotiation.cc:299] Negotiation successful
0924 15:30:03.163002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:30:03.163460 18745 heartbeater.cc:347] Connected to a master server at 127.5.98.254:33037
I20260924 15:30:03.163563 18745 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.163776 18745 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:03.164467 18745 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:33037 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:30:03.164546 18745 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.164611 18745 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:03.164923 18745 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.165007 18745 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
W20260924 15:30:03.165293 18745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:30:03.168542 18749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:03.160127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.5.98.254:33037)
0924 15:30:03.160478 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:03.160480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:03.160488 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:03.160507 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:03.160508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:03.160521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:03.161371 (+ 850us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:03.161407 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.162129 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.163930 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:03.164213 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:03.165875 (+ 1662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:03.165918 (+ 43us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:03.165919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:03.165919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:03.165922 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:03.167280 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:03.167644 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:03.167645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:03.167645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:03.167730 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:03.167946 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:03.167947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:03.167947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:03.168056 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:03.168226 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:03.168264 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:30:03.168390 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":57,"threads_started":1}
Sep 24 15:30:03 dist-test-slave-4g6f krb5kdc[17271](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263803, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:03.813463 18474 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:03.813645 18474 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:03.814410 18273 ts_manager.cc:194] Registered new tserver with Master: 78a30e6068b44b6bafb32f82f65f7cef (127.5.98.193:44609)
I20260924 15:30:03.815521 18273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:39615
I20260924 15:30:04.002143 18610 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:04.002348 18610 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:04.002668 18273 ts_manager.cc:194] Registered new tserver with Master: 9d151362610d43dbbc8c6369259af19c (127.5.98.194:39681)
I20260924 15:30:04.003232 18273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:55579
Sep 24 15:30:04 dist-test-slave-4g6f krb5kdc[17271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263803, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for HTTP/127.5.98.212@KRBTEST.COM
I20260924 15:30:04.165498 18745 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:04.165661 18745 heartbeater.cc:511] Master 127.5.98.254:33037 requested a full tablet report, sending...
I20260924 15:30:04.166111 18272 ts_manager.cc:194] Registered new tserver with Master: a6838f418cc745ba9684c053d3cae535 (127.5.98.195:36005)
I20260924 15:30:04.166471 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:04.166662 18272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:44447
I20260924 15:30:04.176959 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.168099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.5.98.254:33037)
0924 15:30:04.168558 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.168560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.168568 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:04.168593 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.168594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.168605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.169423 (+ 818us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:04.169467 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.170236 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.171294 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.171590 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.173279 (+ 1689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.173336 (+ 57us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:04.173337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:04.173337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:04.173342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:04.175258 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:04.175708 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:04.175709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:04.175709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:04.175771 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:04.176024 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:04.176025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:04.176025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:04.176169 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:04.176701 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.176747 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.176848 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":69,"threads_started":1}
I20260924 15:30:04.179309 18272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:30:04.181623 18272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:30:04.192478 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.185409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44609 (local address 127.0.0.1:44310)
0924 15:30:04.187776 (+ 2367us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.187810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:04.187869 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:04.188532 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:04.188536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:04.188539 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:04.188612 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.188614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.189519 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.189520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.191136 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.191138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.192182 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.192183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.192254 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.192268 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:30:04.192332 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:30:04.192381 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2338,"thread_start_us":92,"threads_started":1}
I20260924 15:30:04.192914 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.185844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.5.98.193:44609)
0924 15:30:04.188277 (+ 2433us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.188283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.188293 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:04.188330 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.188332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.188349 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.188424 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:04.188698 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.189427 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.191219 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.192015 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.192090 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.192149 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.192790 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.192838 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2397,"thread_start_us":46,"threads_started":1}
I20260924 15:30:04.193861 18408 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1703015b0d4747bd661f38d4380a26 (DEFAULT_TABLE table=test-table [id=b389b0fca0714d439b27b5976fae3188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:04.194252 18408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1703015b0d4747bd661f38d4380a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:04.197670 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.185640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:39681 (local address 127.0.0.1:45342)
0924 15:30:04.187995 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.188019 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:04.188057 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:04.193861 (+ 5804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:04.193862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:04.193864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:04.193913 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.193915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.195206 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.195207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.196370 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.196373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.197430 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.197432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.197500 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.197505 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:30:04.197538 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 15:30:04.197578 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"thread_start_us":71,"threads_started":1}
I20260924 15:30:04.198163 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.186049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:36005 (local address 127.0.0.1:45546)
0924 15:30:04.188127 (+ 2078us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.188135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:04.188171 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:04.189811 (+ 1640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:04.189812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:04.189814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:04.189866 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.189867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.191039 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.191040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.192142 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.192145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.193188 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.193189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.193263 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.193270 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:30:04.193318 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:30:04.193355 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":65,"threads_started":1}
I20260924 15:30:04.198217 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.185776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.5.98.194:39681)
0924 15:30:04.193568 (+ 7792us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.193573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.193587 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:30:04.193627 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.193629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.193646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.193740 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:04.194002 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.195089 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.196493 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.197313 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.197987 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.198058 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.198087 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.198131 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7749,"thread_start_us":68,"threads_started":1}
I20260924 15:30:04.198525 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.186284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.5.98.195:36005)
0924 15:30:04.189592 (+ 3308us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.189597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.189606 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:30:04.189640 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.189642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.189657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.189728 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:04.189943 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.190938 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.192229 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.193013 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.193079 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.193132 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.198377 (+ 5245us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.198438 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3271,"thread_start_us":51,"threads_started":1}
I20260924 15:30:04.198737 18780 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Bootstrap starting.
I20260924 15:30:04.198925 18544 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1703015b0d4747bd661f38d4380a26 (DEFAULT_TABLE table=test-table [id=b389b0fca0714d439b27b5976fae3188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:04.199296 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1703015b0d4747bd661f38d4380a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:04.199450 18780 tablet_bootstrap.cc:654] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:04.199818 18780 log.cc:824] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:04.200133 18659 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1703015b0d4747bd661f38d4380a26 (DEFAULT_TABLE table=test-table [id=b389b0fca0714d439b27b5976fae3188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:04.200446 18659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1703015b0d4747bd661f38d4380a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:04.200659 18780 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: No bootstrap required, opened a new log
I20260924 15:30:04.200726 18780 ts_tablet_manager.cc:1397] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:04.203495 18782 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Bootstrap starting.
I20260924 15:30:04.203612 18780 raft_consensus.cc:348] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.203827 18780 raft_consensus.cc:374] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:04.203872 18780 raft_consensus.cc:729] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a30e6068b44b6bafb32f82f65f7cef, State: Initialized, Role: FOLLOWER
I20260924 15:30:04.204021 18780 consensus_queue.cc:260] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.204169 18782 tablet_bootstrap.cc:654] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:04.204324 18780 ts_tablet_manager.cc:1428] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 15:30:04.204464 18474 heartbeater.cc:503] Master 127.5.98.254:33037 was elected leader, sending a full tablet report...
I20260924 15:30:04.204516 18782 log.cc:824] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:04.205636 18783 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Bootstrap starting.
I20260924 15:30:04.205840 18782 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: No bootstrap required, opened a new log
I20260924 15:30:04.205924 18782 ts_tablet_manager.cc:1397] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:04.206336 18783 tablet_bootstrap.cc:654] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:04.206664 18783 log.cc:824] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:04.207509 18783 tablet_bootstrap.cc:492] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: No bootstrap required, opened a new log
I20260924 15:30:04.207589 18783 ts_tablet_manager.cc:1397] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:04.208684 18782 raft_consensus.cc:348] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.208863 18782 raft_consensus.cc:374] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:04.208920 18782 raft_consensus.cc:729] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d151362610d43dbbc8c6369259af19c, State: Initialized, Role: FOLLOWER
I20260924 15:30:04.209070 18782 consensus_queue.cc:260] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.209367 18782 ts_tablet_manager.cc:1428] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:30:04.209491 18610 heartbeater.cc:503] Master 127.5.98.254:33037 was elected leader, sending a full tablet report...
I20260924 15:30:04.210292 18783 raft_consensus.cc:348] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.210476 18783 raft_consensus.cc:374] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:04.210523 18783 raft_consensus.cc:729] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6838f418cc745ba9684c053d3cae535, State: Initialized, Role: FOLLOWER
I20260924 15:30:04.210654 18783 consensus_queue.cc:260] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.210948 18783 ts_tablet_manager.cc:1428] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 15:30:04.210958 18745 heartbeater.cc:503] Master 127.5.98.254:33037 was elected leader, sending a full tablet report...
W20260924 15:30:04.236640 18611 tablet.cc:2369] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:30:04.293123 18476 tablet.cc:2369] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:30:04.295614 18787 raft_consensus.cc:482] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:04.295776 18787 raft_consensus.cc:504] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.296269 18787 leader_election.cc:288] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a30e6068b44b6bafb32f82f65f7cef (127.5.98.193:44609), a6838f418cc745ba9684c053d3cae535 (127.5.98.195:36005)
I20260924 15:30:04.300493 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.296677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:54457 (local address 127.5.98.193:44609)
0924 15:30:04.296800 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.296819 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.296870 (+ 51us) server_negotiation.cc:407] Connection header received
0924 15:30:04.296973 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.296974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.296987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.297016 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:04.297249 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.297947 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.299461 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.300053 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.300109 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.300121 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.300378 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.300422 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 15:30:04.300822 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.296890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59735 (local address 127.5.98.195:36005)
0924 15:30:04.296944 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.296947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.297087 (+ 140us) server_negotiation.cc:407] Connection header received
0924 15:30:04.297217 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.297219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.297233 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.297269 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:04.297539 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.298260 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.299764 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.300365 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.300422 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.300430 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.300671 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.300713 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:30:04.301134 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.296444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44609 (local address 127.5.98.194:54457)
0924 15:30:04.296802 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.296813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:04.296898 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:04.297100 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:04.297101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:04.297103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:04.297181 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.297183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.298071 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.298071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.299354 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.299357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.300204 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.300205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.300275 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.300279 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:04.300789 (+ 510us) client_negotiation.cc:240] Negotiation successful
0924 15:30:04.300835 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260924 15:30:04.301149 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.296551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:36005 (local address 127.5.98.194:59735)
0924 15:30:04.297011 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.297024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:04.297122 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:04.297353 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:04.297354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:04.297355 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:04.297443 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.297445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.298380 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.298381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.299602 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:04.299606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.300498 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.300499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:04.300583 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.300587 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:04.300953 (+ 366us) client_negotiation.cc:240] Negotiation successful
0924 15:30:04.300999 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260924 15:30:04.301669 18428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1703015b0d4747bd661f38d4380a26" candidate_uuid: "9d151362610d43dbbc8c6369259af19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a30e6068b44b6bafb32f82f65f7cef" is_pre_election: true
I20260924 15:30:04.301703 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1703015b0d4747bd661f38d4380a26" candidate_uuid: "9d151362610d43dbbc8c6369259af19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6838f418cc745ba9684c053d3cae535" is_pre_election: true
I20260924 15:30:04.301864 18428 raft_consensus.cc:2454] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d151362610d43dbbc8c6369259af19c in term 0.
I20260924 15:30:04.301877 18680 raft_consensus.cc:2454] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d151362610d43dbbc8c6369259af19c in term 0.
I20260924 15:30:04.302130 18497 leader_election.cc:302] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d151362610d43dbbc8c6369259af19c, a6838f418cc745ba9684c053d3cae535; no voters:
I20260924 15:30:04.302385 18787 raft_consensus.cc:2781] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:04.302443 18787 raft_consensus.cc:482] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:04.302470 18787 raft_consensus.cc:3037] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:04.303399 18787 raft_consensus.cc:504] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.303550 18787 leader_election.cc:288] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [CANDIDATE]: Term 1 election: Requested vote from peers 78a30e6068b44b6bafb32f82f65f7cef (127.5.98.193:44609), a6838f418cc745ba9684c053d3cae535 (127.5.98.195:36005)
I20260924 15:30:04.303768 18428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1703015b0d4747bd661f38d4380a26" candidate_uuid: "9d151362610d43dbbc8c6369259af19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a30e6068b44b6bafb32f82f65f7cef"
I20260924 15:30:04.303826 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c1703015b0d4747bd661f38d4380a26" candidate_uuid: "9d151362610d43dbbc8c6369259af19c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6838f418cc745ba9684c053d3cae535"
I20260924 15:30:04.303855 18428 raft_consensus.cc:3037] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:04.303903 18680 raft_consensus.cc:3037] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:04.304759 18680 raft_consensus.cc:2454] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d151362610d43dbbc8c6369259af19c in term 1.
I20260924 15:30:04.304841 18428 raft_consensus.cc:2454] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d151362610d43dbbc8c6369259af19c in term 1.
I20260924 15:30:04.305028 18495 leader_election.cc:302] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78a30e6068b44b6bafb32f82f65f7cef, 9d151362610d43dbbc8c6369259af19c; no voters:
I20260924 15:30:04.305212 18787 raft_consensus.cc:2781] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:04.305367 18787 raft_consensus.cc:686] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 1 LEADER]: Becoming Leader. State: Replica: 9d151362610d43dbbc8c6369259af19c, State: Running, Role: LEADER
I20260924 15:30:04.305454 18787 consensus_queue.cc:237] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } }
I20260924 15:30:04.306327 18272 catalog_manager.cc:5495] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d151362610d43dbbc8c6369259af19c (127.5.98.194). New cstate: current_term: 1 leader_uuid: "9d151362610d43dbbc8c6369259af19c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d151362610d43dbbc8c6369259af19c" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 39681 } health_report { overall_health: HEALTHY } } }
I20260924 15:30:04.337515 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.334081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.5.98.194:39681)
0924 15:30:04.334127 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:04.334129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:04.334137 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:04.334180 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:04.334181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:04.334205 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:04.334274 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:04.334490 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.335226 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.336355 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:04.336640 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:04.336720 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:04.336913 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:04.337057 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:04.337148 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:04.337413 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 15:30:04.337436 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:30:04.340018 18428 raft_consensus.cc:1261] T 0c1703015b0d4747bd661f38d4380a26 P 78a30e6068b44b6bafb32f82f65f7cef [term 1 FOLLOWER]: Refusing update from remote peer 9d151362610d43dbbc8c6369259af19c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:04.340052 18680 raft_consensus.cc:1261] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 1 FOLLOWER]: Refusing update from remote peer 9d151362610d43dbbc8c6369259af19c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:04.340448 18791 consensus_queue.cc:1035] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6838f418cc745ba9684c053d3cae535" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:04.340564 18787 consensus_queue.cc:1035] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a30e6068b44b6bafb32f82f65f7cef" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 44609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:04.350730 18797 mvcc.cc:205] Tried to move back new op lower bound from 7332920542572859392 to 7332920542435610624. Current Snapshot: MvccSnapshot[applied={T|T < 7332920542572859392}]
I20260924 15:30:04.351955 18272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:04.352051 18272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:04.353012 18272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f33c950a34e94bbb9e09918332950222: Sending DeleteTablet for 3 replicas of tablet 0c1703015b0d4747bd661f38d4380a26
I20260924 15:30:04.353385 18659 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1703015b0d4747bd661f38d4380a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:04 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:45546
I20260924 15:30:04.353536 18408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1703015b0d4747bd661f38d4380a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:04 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:44310
I20260924 15:30:04.353597 18544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c1703015b0d4747bd661f38d4380a26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:04 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:45342
I20260924 15:30:04.354228 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18312
I20260924 15:30:04.356598 18803 tablet_replica.cc:330] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: stopping tablet replica
I20260924 15:30:04.356742 18803 raft_consensus.cc:2229] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:04.356922 18803 raft_consensus.cc:2258] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:04.357426 18803 ts_tablet_manager.cc:1905] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:04.358201 18805 tablet_replica.cc:330] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: stopping tablet replica
I20260924 15:30:04.358310 18805 raft_consensus.cc:2229] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:30:04.358429 18805 raft_consensus.cc:2258] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:04.359323 18803 ts_tablet_manager.cc:1918] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:04.359450 18803 log.cc:1196] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/0c1703015b0d4747bd661f38d4380a26
I20260924 15:30:04.360023 18803 ts_tablet_manager.cc:1939] T 0c1703015b0d4747bd661f38d4380a26 P a6838f418cc745ba9684c053d3cae535: Deleting consensus metadata
I20260924 15:30:04.360450 18259 catalog_manager.cc:4841] TS a6838f418cc745ba9684c053d3cae535 (127.5.98.195:36005): tablet 0c1703015b0d4747bd661f38d4380a26 (table test-table [id=b389b0fca0714d439b27b5976fae3188]) successfully deleted
W20260924 15:30:04.362550 18257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:44609 (error 108)
I20260924 15:30:04.363045 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:04.362899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:44609 (local address 127.0.0.1:44316)
0924 15:30:04.362964 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:04.363006 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 15:30:04.363152 18257 catalog_manager.cc:4584] TS 78a30e6068b44b6bafb32f82f65f7cef (127.5.98.193:44609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c1703015b0d4747bd661f38d4380a26: Network error: Client connection negotiation failed: client connection to 127.5.98.193:44609: connect: Connection refused (error 111)
I20260924 15:30:04.363420 18805 ts_tablet_manager.cc:1905] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:04.364967 18805 ts_tablet_manager.cc:1918] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:04.365056 18805 log.cc:1196] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/0c1703015b0d4747bd661f38d4380a26
I20260924 15:30:04.365329 18805 ts_tablet_manager.cc:1939] T 0c1703015b0d4747bd661f38d4380a26 P 9d151362610d43dbbc8c6369259af19c: Deleting consensus metadata
I20260924 15:30:04.365705 18257 catalog_manager.cc:4841] TS 9d151362610d43dbbc8c6369259af19c (127.5.98.194:39681): tablet 0c1703015b0d4747bd661f38d4380a26 (table test-table [id=b389b0fca0714d439b27b5976fae3188]) successfully deleted
I20260924 15:30:04.367825 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18475
I20260924 15:30:04.374609 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18613
I20260924 15:30:04.381488 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18238
I20260924 15:30:04.387130 5515 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:30:04.731247 5515 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:30:04.731335 5515 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:30:05.075639 5515 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:30:05.075 UTC [17301] LOG: received smart shutdown request
2026-09-24 15:30:05.080 UTC [17301] LOG: background worker "logical replication launcher" (PID 17310) exited with exit code 1
2026-09-24 15:30:05.080 UTC [17305] LOG: shutting down
2026-09-24 15:30:05.247 UTC [17301] LOG: database system is shut down
2026-09-24T15:30:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47253 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:30:05 dist-test-slave-4g6f krb5kdc[18816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:30:05 dist-test-slave-4g6f krb5kdc[18816](info): set up 2 sockets
Sep 24 15:30:05 dist-test-slave-4g6f krb5kdc[18816](info): commencing operation
krb5kdc: starting...
W20260924 15:30:06.408855 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:30:06.524672 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36013
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:06.605209 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:06.605414 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:06.605436 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:06.606484 18830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:06.606518 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:06.606532 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:06.606544 18830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:06.606556 18830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:06.608095 18830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:06.608289 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:06.608522 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:06.613282 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.613300 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.613302 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:06.613611 18830 server_base.cc:1034] running on GCE node
I20260924 15:30:06.614117 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:06.614401 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:06.615566 18830 hybrid_clock.cc:648] HybridClock initialized: now 1790263806615560 us; error 36 us; skew 500 ppm
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:06.621105 18830 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:06.621599 18830 webserver.cc:460] Webserver started at http://127.5.98.254:34991/ using document root <none> and password file <none>
I20260924 15:30:06.621826 18830 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:06.621894 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:06.622061 18830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:06.623124 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "b7e632e3670541bdadd2546155921be8"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "b7ad3cda3475fcbc70456088fa43fa0e"
server_key_iv: "038ba984b5a40b359abc72fa29bb0f1d"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.623503 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "b7e632e3670541bdadd2546155921be8"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "b7ad3cda3475fcbc70456088fa43fa0e"
server_key_iv: "038ba984b5a40b359abc72fa29bb0f1d"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.624871 18830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:06.625666 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:06.625864 18830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:06.625964 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "b7e632e3670541bdadd2546155921be8"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "b7ad3cda3475fcbc70456088fa43fa0e"
server_key_iv: "038ba984b5a40b359abc72fa29bb0f1d"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.626065 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:06.651008 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:06.653358 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:06.653471 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:06.657878 18830 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36013
I20260924 15:30:06.657948 18899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36013 every 8 connection(s)
I20260924 15:30:06.658309 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:06.659140 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18830
I20260924 15:30:06.659183 18900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:06.659273 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:30:06.659458 5515 external_mini_cluster.cc:1428] Setting key 9d8716f01e5fd6965a6f4aa2d069d024
I20260924 15:30:06.661165 18900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.98.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.253@KRBTEST.COM" created.
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:06.673660 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.662043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36013 (local address 127.0.0.1:38890)
0924 15:30:06.662210 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.662247 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.662766 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.663098 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.663101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.663378 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.663530 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.663535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.664614 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.664615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.665280 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.665282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.665801 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.665802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.665868 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.666471 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.666480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.670205 (+ 3725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.672080 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.672087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.672089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.672312 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.672602 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.672603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.672603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.672686 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.673141 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.673147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.673216 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.673295 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.673485 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260924 15:30:06.673772 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.662241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.5.98.254:36013)
0924 15:30:06.662363 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.662368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.662403 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:30:06.662978 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.662980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.662997 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.663093 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.663642 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.664498 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.665377 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.665696 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.665826 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.670315 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.670323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.670323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.670338 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.671997 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.672406 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.672410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.672410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.672520 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.672754 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.672755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.672756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.673064 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.673157 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.673366 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.673609 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260924 15:30:06.674533 18900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:30:06.674944 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.674794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55600)
0924 15:30:06.674851 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.674909 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:30:06.675115 18849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
I20260924 15:30:06.675621 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.675502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55606)
0924 15:30:06.675561 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.675594 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 15:30:06.675837 18900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:46271: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
I20260924 15:30:06.684768 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:46271
--webserver_interface=127.5.98.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:06.725098 18900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } attempt: 1
I20260924 15:30:06.725661 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.725497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55622)
0924 15:30:06.725577 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.725626 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 15:30:06.726059 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.725965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55626)
0924 15:30:06.726013 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.726040 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:30:06.726152 18900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:46271: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
W20260924 15:30:06.766986 18907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:06.767198 18907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:06.767220 18907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:06.768306 18907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:06.768342 18907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:06.768355 18907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:06.768368 18907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:06.768380 18907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:06.769788 18907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.98.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:06.769984 18907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:06.770155 18907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:06.774991 18912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.774971 18913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.774988 18915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:06.775177 18907 server_base.cc:1034] running on GCE node
I20260924 15:30:06.775543 18907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:06.775884 18907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:06.777066 18907 hybrid_clock.cc:648] HybridClock initialized: now 1790263806777040 us; error 39 us; skew 500 ppm
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:06.779585 18907 init.cc:377] Logged in from keytab as kudu/127.5.98.253@KRBTEST.COM (short username kudu)
I20260924 15:30:06.780092 18907 webserver.cc:460] Webserver started at http://127.5.98.253:43557/ using document root <none> and password file <none>
I20260924 15:30:06.780283 18907 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:06.780328 18907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:06.780417 18907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:06.781417 18907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data/instance:
uuid: "f7850254bffa40ca9fc84f0b5f798e94"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "50c0b6417116e370bec8c7f8d1ed92ef"
server_key_iv: "29ca57db6517989f35b99822aad444f1"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.781749 18907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal/instance:
uuid: "f7850254bffa40ca9fc84f0b5f798e94"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "50c0b6417116e370bec8c7f8d1ed92ef"
server_key_iv: "29ca57db6517989f35b99822aad444f1"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.783106 18907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:30:06.783859 18922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:06.784024 18907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:06.784132 18907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "f7850254bffa40ca9fc84f0b5f798e94"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "50c0b6417116e370bec8c7f8d1ed92ef"
server_key_iv: "29ca57db6517989f35b99822aad444f1"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.784209 18907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:06.794294 18900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } attempt: 2
I20260924 15:30:06.794765 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.794638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55630)
0924 15:30:06.794698 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.794737 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:30:06.795167 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.795066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55642)
0924 15:30:06.795114 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.795146 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:30:06.795248 18900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:46271: Network error: Client connection negotiation failed: client connection to 127.5.98.253:46271: connect: Connection refused (error 111)
I20260924 15:30:06.810420 18907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:06.812507 18907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:06.812709 18907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:06.816967 18907 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.253:46271
I20260924 15:30:06.817029 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.253:46271 every 8 connection(s)
I20260924 15:30:06.817387 18907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
I20260924 15:30:06.818284 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:06.819909 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18907
I20260924 15:30:06.820019 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-1/wal/instance
I20260924 15:30:06.819907 18977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:30:06.820218 5515 external_mini_cluster.cc:1428] Setting key 7aea9c6b5b3cc95a94e2edd2fbc7b8c5
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.252@KRBTEST.COM; defaulting to no policy
I20260924 15:30:06.829794 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.820791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.5.98.254:36013)
0924 15:30:06.820868 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.820870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.821081 (+ 211us) server_negotiation.cc:407] Connection header received
0924 15:30:06.821608 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.821610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.821627 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.821693 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.822439 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.822946 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.824101 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.824334 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.824433 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.826633 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.826635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.826636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.826647 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.828208 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.828801 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.828802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.828803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.828876 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.829142 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.829143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.829143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.829250 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.829311 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.829612 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.829710 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:30:06.829995 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.820760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36013 (local address 127.0.0.1:38904)
0924 15:30:06.820940 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.820985 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.821484 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.821766 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.821770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.822107 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.822320 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.822325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.823065 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.823066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.823996 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.823998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.824456 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.824457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.824593 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.825135 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.825143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.826525 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.828341 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.828349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.828350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.828703 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.828972 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.828973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.828973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.829088 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.829308 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.829313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.829455 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.829568 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.829735 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.252@KRBTEST.COM" created.
I20260924 15:30:06.830601 18977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:30:06.838635 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.830911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55650)
0924 15:30:06.830966 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.830980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.831028 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.831289 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.831290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.831455 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.831516 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.831517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.832920 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.832921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.833416 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.833418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.833940 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.833941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.834013 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.834431 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.834434 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.835642 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.837484 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.837485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.837486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.837658 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.837895 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.837896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.837896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.837955 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.838282 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.838283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.838323 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.838401 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.838500 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:30:06.838726 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.830968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.5.98.253:46271)
0924 15:30:06.831129 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.831133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.831153 (+ 20us) server_negotiation.cc:407] Connection header received
0924 15:30:06.831178 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.831180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.831200 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.831270 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.831610 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.832819 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.833525 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.833845 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.833941 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.835765 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.835775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.835775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.835782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.837376 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.837757 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.837760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.837760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.837824 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.838031 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.838032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.838033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.838191 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.838310 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.838433 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.838572 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260924 15:30:06.839468 18977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:30:06.839876 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.839736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60520)
0924 15:30:06.839793 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.839849 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:30:06.840018 18925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
I20260924 15:30:06.840507 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.840425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60532)
0924 15:30:06.840466 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.840490 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:30:06.840641 18977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:42747: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
I20260924 15:30:06.844691 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:42747
--webserver_interface=127.5.98.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:06.886430 18900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } attempt: 3
I20260924 15:30:06.889851 18977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } attempt: 1
I20260924 15:30:06.890235 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.890103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60544)
0924 15:30:06.890160 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.890197 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:30:06.890594 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.890468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60556)
0924 15:30:06.890554 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.890574 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 15:30:06.890694 18977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:42747: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:30:06.894018 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.886903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.5.98.253:46271)
0924 15:30:06.886982 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.886984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.886998 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:30:06.887080 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.887081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.887095 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.887160 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.887474 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.888276 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.888965 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.889274 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.889349 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.891547 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.891549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.891549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.891555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.892982 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.893358 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.893358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.893359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.893407 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.893593 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.893594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.893594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.893696 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.893745 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.893903 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.893965 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:30:06.894073 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.886848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55656)
0924 15:30:06.886928 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.886943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.887008 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.887174 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.887174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.887325 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.887393 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.887395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.888358 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.888359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.888876 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.888877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.889352 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.889353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.889411 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.889894 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.889900 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.891451 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.893065 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.893066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.893067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.893282 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.893470 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.893471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.893471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.893535 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.893756 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.893757 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.893807 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.893879 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.893978 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 15:30:06.894541 18900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:30:06.894840 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.894756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60558)
0924 15:30:06.894788 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.894817 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 15:30:06.895125 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.895070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60560)
0924 15:30:06.895092 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.895112 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 15:30:06.895223 18900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:42747: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
I20260924 15:30:06.926380 18900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } attempt: 1
I20260924 15:30:06.926956 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.926783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60574)
0924 15:30:06.926864 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.926916 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 15:30:06.927371 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.927267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60590)
0924 15:30:06.927315 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.927342 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:30:06.927461 18900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:42747: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
W20260924 15:30:06.928946 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:06.929136 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:06.929183 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:06.930390 18984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:06.930434 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:06.930449 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:06.930498 18984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:06.930524 18984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:06.932077 18984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36013,127.5.98.253:46271,127.5.98.252:42747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.98.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:06.932340 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:06.932574 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:06.937312 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.937340 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:06.937335 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:06.937603 18984 server_base.cc:1034] running on GCE node
I20260924 15:30:06.937876 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:06.938145 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:06.939284 18984 hybrid_clock.cc:648] HybridClock initialized: now 1790263806939267 us; error 28 us; skew 500 ppm
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:06.941787 18984 init.cc:377] Logged in from keytab as kudu/127.5.98.252@KRBTEST.COM (short username kudu)
I20260924 15:30:06.942261 18984 webserver.cc:460] Webserver started at http://127.5.98.252:34405/ using document root <none> and password file <none>
I20260924 15:30:06.942451 18984 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:06.942495 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:06.942588 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:06.943599 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data/instance:
uuid: "622ee607597d44d39a82115db00992c4"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "3aeff90730239fc027dca308ad3b4543"
server_key_iv: "a757a3a51c8389672e2461cf6b8e2eb0"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.944008 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal/instance:
uuid: "622ee607597d44d39a82115db00992c4"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "3aeff90730239fc027dca308ad3b4543"
server_key_iv: "a757a3a51c8389672e2461cf6b8e2eb0"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.945346 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:06.946018 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:06.946211 18984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:06.946296 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "622ee607597d44d39a82115db00992c4"
format_stamp: "Formatted at 2026-09-24 15:30:06 on dist-test-slave-4g6f"
server_key: "3aeff90730239fc027dca308ad3b4543"
server_key_iv: "a757a3a51c8389672e2461cf6b8e2eb0"
server_key_version: "encryptionkey@0"
I20260924 15:30:06.946353 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:06.958827 18977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } attempt: 2
I20260924 15:30:06.959270 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.959125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60598)
0924 15:30:06.959189 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.959232 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:30:06.959630 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.959550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60604)
0924 15:30:06.959591 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.959614 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:30:06.959762 18977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:42747: Network error: Client connection negotiation failed: client connection to 127.5.98.252:42747: connect: Connection refused (error 111)
I20260924 15:30:06.965222 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:06.967093 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:06.967242 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:06.971136 18984 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.252:42747
I20260924 15:30:06.971184 19053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.252:42747 every 8 connection(s)
I20260924 15:30:06.971540 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
I20260924 15:30:06.972448 19054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:06.973933 19054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:06.979312 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 18984
I20260924 15:30:06.979446 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263655576751-5515-0/minicluster-data/master-2/wal/instance
I20260924 15:30:06.979693 5515 external_mini_cluster.cc:1428] Setting key 10c5d32d1a09b5ea0df6892287116f69
I20260924 15:30:06.982952 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.974609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.5.98.254:36013)
0924 15:30:06.974669 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.974671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.974848 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:30:06.975286 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.975287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.975296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.975347 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.975985 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.976463 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.977442 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.977626 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.977676 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.979807 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.979809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.979810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.979823 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.981572 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.981989 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.981991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.981991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.982054 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.982310 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.982311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.982311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.982446 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.982513 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.982764 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.982860 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:30:06.982977 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.974576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36013 (local address 127.0.0.1:38912)
0924 15:30:06.974728 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.974768 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.975206 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.975411 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.975414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.975641 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.975892 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.975898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.976561 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.976563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.977337 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.977339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.977697 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.977698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.977803 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.978208 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.978216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.979721 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.981656 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.981663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.981669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.981889 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.982130 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.982131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.982131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.982235 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.982513 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.982519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.982633 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.982717 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.982828 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260924 15:30:06.983493 19054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:30:06.988873 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.980073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.5.98.252:42747)
0924 15:30:06.980226 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.980231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.980581 (+ 350us) server_negotiation.cc:407] Connection header received
0924 15:30:06.980610 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.980611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.980622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.980729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.981474 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.982272 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.983269 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.983630 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.983759 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.985864 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.985872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.985872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.985878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.987564 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.987917 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.987921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.987921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.987980 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.988196 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.988196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.988197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.988391 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.988559 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.988649 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.988749 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260924 15:30:06.990453 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.983866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.5.98.253:46271)
0924 15:30:06.983934 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.983937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.983945 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:06.983969 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.983970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.983978 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.984044 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.984307 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.984758 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.985404 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.985642 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.985717 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.987690 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.987691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.987691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.987699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.989273 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.989625 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.989626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.989626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.989675 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.989930 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.989931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.989931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.990087 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.990151 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.990332 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.990393 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:30:06.990526 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.983762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:46271 (local address 127.0.0.1:55664)
0924 15:30:06.983835 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.983845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.983902 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.984052 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.984053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.984180 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.984221 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.984222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.984835 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.984836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.985319 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.985320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.985715 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.985716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.985787 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.986142 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.986147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.987570 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.989351 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.989352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.989352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.989551 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.989788 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.989789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.989789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.989862 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.990165 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.990166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.990229 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.990311 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.990416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 15:30:06.990955 19054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:30:06.997422 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.990190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.5.98.254:36013)
0924 15:30:06.990245 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.990247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.990255 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:06.990276 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.990277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.990288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.990337 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.991048 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.991589 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.992280 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.992482 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.992557 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.994623 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.994624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.994624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.994632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.996270 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.996612 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.996613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.996613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.996674 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.996877 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.996878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.996878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.996989 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.997197 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.997235 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.997309 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:30:06.998059 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.991226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60620)
0924 15:30:06.991265 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:06.991273 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:06.991305 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:06.991420 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:06.991420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:06.991546 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:06.991570 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.991572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.992229 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.992230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.992685 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:06.992686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.993240 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.993241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:06.993290 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.993668 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:06.993672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:06.994954 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:06.996790 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.996791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.996791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.996964 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.997190 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:06.997191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:06.997192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:06.997248 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:06.997681 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:06.997682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:06.997715 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:30:06.997823 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:30:06.997932 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:30:06.998162 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.991279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.5.98.252:42747)
0924 15:30:06.991312 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.991314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.991323 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:30:06.991358 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.991358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.991366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.991417 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.991643 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.992139 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.992966 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.993150 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:06.993238 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:06.995035 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:06.995037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:06.995037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:06.995045 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:06.996673 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.997056 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.997057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.997058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.997111 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:06.997431 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:06.997432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:06.997432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:06.997591 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:06.997657 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:06.997856 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:30:06.997992 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":20}
I20260924 15:30:06.999666 19054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: Bootstrap starting.
I20260924 15:30:07.000645 19054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:07.000960 19054 log.cc:824] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:07.001829 19054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: No bootstrap required, opened a new log
I20260924 15:30:07.002607 18900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } attempt: 2
Sep 24 15:30:06 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:30:07.004020 19054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.004149 19054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:07.004223 19054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622ee607597d44d39a82115db00992c4, State: Initialized, Role: FOLLOWER
I20260924 15:30:07.004345 19054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.004654 19054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:07.006155 19060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.006276 19060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:30:07.006765 19071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:30:07.006863 19071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:30:07.007207 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:06.998109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.5.98.253:46271)
0924 15:30:06.998587 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:06.998590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:06.998598 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:06.998622 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:06.998623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:06.998632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:06.998686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:06.999154 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:06.999848 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.000502 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.000799 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.000869 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.003320 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.003321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.003321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.003329 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.005780 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.006190 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.006191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.006192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.006326 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.006561 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.006562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.006562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.006696 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.006876 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.006985 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.007096 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456}
Sep 24 15:30:07 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:30:07.008706 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:30:07.010829 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.002910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.5.98.252:42747)
0924 15:30:07.002945 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:07.002946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:07.002957 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:30:07.002978 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:07.002979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:07.002987 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:07.003065 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:07.003344 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.004116 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.005350 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.005656 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.005726 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.007923 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.007924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.007925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.007931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.009524 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.009933 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.009935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.009935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.009992 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.010249 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.010251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.010251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.010419 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.010495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.010670 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.010748 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:30:07.010835 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.002850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60628)
0924 15:30:07.002887 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:07.002898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:07.002936 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:07.003072 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:07.003073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:07.003209 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:07.003242 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:07.003243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.004214 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.004215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:07.005246 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:07.005251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.005782 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.005783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:07.005846 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.006447 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:07.006452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:07.007857 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:07.009641 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:07.009642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:07.009643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:07.009850 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:07.010097 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:07.010098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:07.010099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:07.010168 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:07.010504 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:07.010505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:07.010550 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:30:07.010626 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:30:07.010722 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 15:30:07.008993 18984 master_runner.cc:186] Error getting master registration for 127.5.98.254:36013: OK, instance_id { permanent_uuid: "b7e632e3670541bdadd2546155921be8" instance_seqno: 1790263806656662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:30:07.012324 18984 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:30:07.012880 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: Bootstrap starting.
I20260924 15:30:07.013607 18900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:07.013924 18900 log.cc:824] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:07.014740 18900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: No bootstrap required, opened a new log
I20260924 15:30:07.016804 18900 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.016953 18900 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:07.016990 18900 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e632e3670541bdadd2546155921be8, State: Initialized, Role: FOLLOWER
I20260924 15:30:07.017110 18900 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.017414 18900 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:07.019928 18830 master_runner.cc:186] Error getting master registration for 127.5.98.253:46271: OK, instance_id { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" instance_seqno: 1790263806815770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:30:07.022678 18830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:30:07.023088 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.023470 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:30:07.024077 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.016050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.5.98.254:36013)
0924 15:30:07.016091 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:07.016093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:07.016210 (+ 117us) server_negotiation.cc:407] Connection header received
0924 15:30:07.016453 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:07.016454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:07.016466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:07.016497 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:07.017188 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.017918 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.019056 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.019332 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.019371 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.020886 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.020888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.020888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.020895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.022547 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.023017 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.023018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.023019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.023080 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.023297 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.023298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.023298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.023426 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.023515 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.023900 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.023988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:30:07.027873 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.016288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.5.98.253:46271)
0924 15:30:07.016334 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:07.016336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:07.017280 (+ 944us) server_negotiation.cc:407] Connection header received
0924 15:30:07.017463 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:07.017464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:07.017473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:07.017501 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:07.019016 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.019704 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.022530 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.022830 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.022874 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.025046 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.025047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.025048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.025054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.026635 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.026995 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.026996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.026996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.027057 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.027268 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.027269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.027269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.027378 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.027500 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.027649 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.027759 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:30:07.028209 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.016466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.5.98.252:42747)
0924 15:30:07.016502 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:07.016504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:07.017369 (+ 865us) server_negotiation.cc:407] Connection header received
0924 15:30:07.018236 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:07.018237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:07.018247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:07.018276 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:07.020767 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.021478 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.023228 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.023525 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.023584 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.025366 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.025367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.025367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.025374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.026940 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.027317 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.027318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.027318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.027378 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.027563 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.027564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.027565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.027724 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.027816 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.028039 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.028119 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:30:07.028496 19092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:30:07.028550 19092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:30:07.050935 18977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } attempt: 3
Sep 24 15:30:07 dist-test-slave-4g6f krb5kdc[18816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263806, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:30:07.059140 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.051279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.5.98.252:42747)
0924 15:30:07.051318 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:07.051320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:07.051328 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:07.051352 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:07.051353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:07.051362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:07.051431 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:07.051767 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.052519 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.053511 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.053815 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.053886 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.056233 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:07.056235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:07.056235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:07.056243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:07.057906 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.058316 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.058318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.058318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.058385 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:07.058601 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:07.058601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:07.058602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:07.058733 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:07.058801 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:07.058966 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 15:30:07.059042 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:30:07.059154 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:07.051209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:42747 (local address 127.0.0.1:60640)
0924 15:30:07.051258 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:07.051269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:07.051305 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:07.051436 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:07.051437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:07.051616 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:07.051691 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:07.051694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.052617 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.052619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:07.053425 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:07.053427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:07.053895 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:07.053896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:07.053961 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:07.054438 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:07.054444 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:07.056128 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:07.058008 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:07.058009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:07.058009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:07.058234 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:07.058461 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:07.058462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:07.058463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:07.058531 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:07.058805 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:07.058806 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:07.058855 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:30:07.058929 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:30:07.059044 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:30:07.061076 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: Bootstrap starting.
I20260924 15:30:07.061872 18977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:07.062196 18977 log.cc:824] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:07.063222 18977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: No bootstrap required, opened a new log
I20260924 15:30:07.065255 18977 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.065407 18977 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:07.065442 18977 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7850254bffa40ca9fc84f0b5f798e94, State: Initialized, Role: FOLLOWER
I20260924 15:30:07.065583 18977 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.065901 18977 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:07.065864 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.066048 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:30:07.067728 19106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:30:07.067828 19106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:30:07.069769 18907 master_runner.cc:263] No leader master found from master f7850254bffa40ca9fc84f0b5f798e94
I20260924 15:30:07.069823 18907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:30:07.095042 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:07.095181 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.095434 19081 leader_election.cc:288] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7850254bffa40ca9fc84f0b5f798e94 (127.5.98.253:46271), 622ee607597d44d39a82115db00992c4 (127.5.98.252:42747)
I20260924 15:30:07.095767 19027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7e632e3670541bdadd2546155921be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622ee607597d44d39a82115db00992c4" is_pre_election: true
I20260924 15:30:07.095942 19027 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e632e3670541bdadd2546155921be8 in term 0.
I20260924 15:30:07.096240 18950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7e632e3670541bdadd2546155921be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7850254bffa40ca9fc84f0b5f798e94" is_pre_election: true
I20260924 15:30:07.096410 18950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e632e3670541bdadd2546155921be8 in term 0.
I20260924 15:30:07.096596 18849 leader_election.cc:302] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e632e3670541bdadd2546155921be8, f7850254bffa40ca9fc84f0b5f798e94; no voters:
I20260924 15:30:07.096865 19081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:07.096930 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:07.096961 19081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:07.097836 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.097957 19081 leader_election.cc:288] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [CANDIDATE]: Term 1 election: Requested vote from peers f7850254bffa40ca9fc84f0b5f798e94 (127.5.98.253:46271), 622ee607597d44d39a82115db00992c4 (127.5.98.252:42747)
I20260924 15:30:07.098203 19027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7e632e3670541bdadd2546155921be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622ee607597d44d39a82115db00992c4"
I20260924 15:30:07.098282 19027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:07.099117 19027 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e632e3670541bdadd2546155921be8 in term 1.
I20260924 15:30:07.099218 18950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7e632e3670541bdadd2546155921be8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7850254bffa40ca9fc84f0b5f798e94"
I20260924 15:30:07.099291 18950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:07.099792 18849 leader_election.cc:302] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622ee607597d44d39a82115db00992c4, b7e632e3670541bdadd2546155921be8; no voters:
I20260924 15:30:07.100425 18950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e632e3670541bdadd2546155921be8 in term 1.
I20260924 15:30:07.100611 19081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:07.100773 19081 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [term 1 LEADER]: Becoming Leader. State: Replica: b7e632e3670541bdadd2546155921be8, State: Running, Role: LEADER
I20260924 15:30:07.100899 19081 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } }
I20260924 15:30:07.101262 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e632e3670541bdadd2546155921be8. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.101363 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:07.101610 19110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:07.101799 19110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:07.103734 19027 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [term 1 FOLLOWER]: Refusing update from remote peer b7e632e3670541bdadd2546155921be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:07.104017 19081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:07.105288 19060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e632e3670541bdadd2546155921be8. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.105405 19060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:30:07.106451 18950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [term 1 FOLLOWER]: Refusing update from remote peer b7e632e3670541bdadd2546155921be8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:07.106695 19081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:07.108011 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e632e3670541bdadd2546155921be8. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.108104 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:30:07.109477 19111 mvcc.cc:205] Tried to move back new op lower bound from 7332920553894514688 to 7332920553885859840. Current Snapshot: MvccSnapshot[applied={T|T < 7332920553894514688}]
I20260924 15:30:07.109959 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.110045 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:07.110415 19060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.110514 19060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:30:07.111135 19112 mvcc.cc:205] Tried to move back new op lower bound from 7332920553894514688 to 7332920553885859840. Current Snapshot: MvccSnapshot[applied={T|T < 7332920553894514688}]
I20260924 15:30:07.111656 19110 catalog_manager.cc:1347] Generated new cluster ID: dbf65b115e584b478fd028aef9a84b62
I20260924 15:30:07.111725 19110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:07.116559 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.116644 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:07.117058 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7e632e3670541bdadd2546155921be8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7e632e3670541bdadd2546155921be8" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36013 } } peers { permanent_uuid: "f7850254bffa40ca9fc84f0b5f798e94" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 46271 } } peers { permanent_uuid: "622ee607597d44d39a82115db00992c4" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 42747 } } }
I20260924 15:30:07.117159 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:30:07.119740 19114 mvcc.cc:205] Tried to move back new op lower bound from 7332920553894514688 to 7332920553885859840. Current Snapshot: MvccSnapshot[applied={T|T < 7332920553894514688}]
I20260924 15:30:07.123261 19110 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:30:07.124020 19110 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:07.129066 19110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e632e3670541bdadd2546155921be8: Generated new TSK 0
I20260924 15:30:07.129236 19110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:30:08.007463 19071 catalog_manager.cc:1259] Loaded cluster ID: dbf65b115e584b478fd028aef9a84b62
I20260924 15:30:08.007550 19071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: loading cluster ID for follower catalog manager: success
I20260924 15:30:08.008421 19071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: acquiring CA information for follower catalog manager: success
I20260924 15:30:08.008718 19071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 622ee607597d44d39a82115db00992c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:30:08.068289 19106 catalog_manager.cc:1259] Loaded cluster ID: dbf65b115e584b478fd028aef9a84b62
I20260924 15:30:08.068367 19106 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: loading cluster ID for follower catalog manager: success
I20260924 15:30:08.069150 19106 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: acquiring CA information for follower catalog manager: success
I20260924 15:30:08.069408 19106 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7850254bffa40ca9fc84f0b5f798e94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:30:08.185184 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18830
I20260924 15:30:08.190958 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18907
I20260924 15:30:08.196049 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 18984
2026-09-24T15:30:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2834 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:30:08 dist-test-slave-4g6f krb5kdc[19130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:30:08 dist-test-slave-4g6f krb5kdc[19130](info): set up 2 sockets
Sep 24 15:30:08 dist-test-slave-4g6f krb5kdc[19130](info): commencing operation
krb5kdc: starting...
W20260924 15:30:09.244783 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:30:09.362375 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34981
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:09.442749 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:09.442986 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:09.443020 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:09.444242 19145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:09.444303 19145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:09.444327 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:09.444357 19145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:09.444409 19145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:09.445856 19145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:09.446130 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:09.446365 19145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:09.451296 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:09.451378 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:09.451396 19145 server_base.cc:1034] running on GCE node
W20260924 15:30:09.451296 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:09.451844 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:09.452198 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:09.453346 19145 hybrid_clock.cc:648] HybridClock initialized: now 1790263809453325 us; error 40 us; skew 500 ppm
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263809, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:09.455982 19145 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:09.456423 19145 webserver.cc:460] Webserver started at http://127.5.98.254:37185/ using document root <none> and password file <none>
I20260924 15:30:09.456678 19145 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:09.456764 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:09.456873 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:09.457890 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "02dc4582bd7540c18764503c729535bd"
format_stamp: "Formatted at 2026-09-24 15:30:09 on dist-test-slave-4g6f"
server_key: "0e73ebba6e12851b12f7e46ba7b5a768"
server_key_iv: "1e98e358c9edd27ccd8c73671bc2b950"
server_key_version: "encryptionkey@0"
I20260924 15:30:09.458204 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "02dc4582bd7540c18764503c729535bd"
format_stamp: "Formatted at 2026-09-24 15:30:09 on dist-test-slave-4g6f"
server_key: "0e73ebba6e12851b12f7e46ba7b5a768"
server_key_iv: "1e98e358c9edd27ccd8c73671bc2b950"
server_key_version: "encryptionkey@0"
I20260924 15:30:09.459483 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:30:09.460247 19160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:09.460424 19145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:09.460532 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "02dc4582bd7540c18764503c729535bd"
format_stamp: "Formatted at 2026-09-24 15:30:09 on dist-test-slave-4g6f"
server_key: "0e73ebba6e12851b12f7e46ba7b5a768"
server_key_iv: "1e98e358c9edd27ccd8c73671bc2b950"
server_key_version: "encryptionkey@0"
I20260924 15:30:09.460611 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:09.468214 19145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:09.470453 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:09.470630 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:09.474650 19145 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34981
I20260924 15:30:09.474684 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34981 every 8 connection(s)
I20260924 15:30:09.475028 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:09.475963 19213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:09.477134 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19145
I20260924 15:30:09.477242 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:30:09.477430 5515 external_mini_cluster.cc:1428] Setting key 2459c1904438af3138ddce418d9f8d42
I20260924 15:30:09.478565 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: Bootstrap starting.
I20260924 15:30:09.479305 19213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:09.479727 19213 log.cc:824] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:09.480465 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: No bootstrap required, opened a new log
I20260924 15:30:09.482904 19213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } }
I20260924 15:30:09.483086 19213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:09.483153 19213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dc4582bd7540c18764503c729535bd, State: Initialized, Role: FOLLOWER
I20260924 15:30:09.483286 19213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } }
I20260924 15:30:09.483388 19213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:09.483429 19213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:09.483469 19213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:09.484193 19213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } }
I20260924 15:30:09.484330 19213 leader_election.cc:302] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02dc4582bd7540c18764503c729535bd; no voters:
I20260924 15:30:09.484505 19213 leader_election.cc:288] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:30:09.484688 19219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:09.484772 19213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:09.484907 19219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [term 1 LEADER]: Becoming Leader. State: Replica: 02dc4582bd7540c18764503c729535bd, State: Running, Role: LEADER
I20260924 15:30:09.485044 19219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } }
I20260924 15:30:09.485441 19219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02dc4582bd7540c18764503c729535bd. Latest consensus state: current_term: 1 leader_uuid: "02dc4582bd7540c18764503c729535bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } } }
I20260924 15:30:09.485543 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02dc4582bd7540c18764503c729535bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02dc4582bd7540c18764503c729535bd" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34981 } } }
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:09.485605 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:09.485569 19219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd [sys.catalog]: This master's current role is: LEADER
W20260924 15:30:09.486801 19233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:30:09.487084 19233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:30:09.487165 19227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:09.487300 19227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:09.488837 19227 catalog_manager.cc:1347] Generated new cluster ID: 5192d7a6512642e3bbc6eca025c451f2
I20260924 15:30:09.488888 19227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:09.490396 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:09.477956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.5.98.254:34981)
0924 15:30:09.478142 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:09.478150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:09.478175 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:30:09.478570 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:09.478576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:09.478621 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:09.478771 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:09.479651 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:09.480958 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:09.482014 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:09.482645 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:09.482756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:09.486220 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:09.486225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:09.486225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:09.486238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:09.488808 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:09.489212 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:09.489215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:09.489215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:09.489318 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:09.489535 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:09.489536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:09.489537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:09.489770 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:09.489944 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:09.490039 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 15:30:09.490177 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260924 15:30:09.492571 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:30:09.505280 19227 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:30:09.505981 19227 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:09.514729 19227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02dc4582bd7540c18764503c729535bd: Generated new TSK 0
I20260924 15:30:09.514915 19227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:30:09.546618 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19145
2026-09-24T15:30:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1349 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19240](info): set up 2 sockets
Sep 24 15:30:09 dist-test-slave-4g6f krb5kdc[19240](info): commencing operation
krb5kdc: starting...
W20260924 15:30:10.592700 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:30:10 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:30:10.708830 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:10.787621 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:10.787860 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:10.787901 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:10.789026 19253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:10.789100 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:10.789125 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:10.789153 19253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:10.789186 19253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:10.790620 19253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:10.790879 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:10.791111 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:10.795893 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:10.795895 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:10.795872 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:10.796079 19253 server_base.cc:1034] running on GCE node
I20260924 15:30:10.796389 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:10.796649 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:10.797845 19253 hybrid_clock.cc:648] HybridClock initialized: now 1790263810797829 us; error 29 us; skew 500 ppm
Sep 24 15:30:10 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263810, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:10.800663 19253 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:10.801074 19253 webserver.cc:460] Webserver started at http://127.5.98.254:40701/ using document root <none> and password file <none>
I20260924 15:30:10.801280 19253 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:10.801326 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:10.801416 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:10.802471 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "06913431cd9d4f04bb26a0a654ea16b1"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "8b59e8093e8fd289f1b1f30a48488c45"
server_key_iv: "e867212ef542122800087151fa5f647d"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.802874 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "06913431cd9d4f04bb26a0a654ea16b1"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "8b59e8093e8fd289f1b1f30a48488c45"
server_key_iv: "e867212ef542122800087151fa5f647d"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.804281 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:30:10.804962 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:10.805145 19253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:10.805261 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "06913431cd9d4f04bb26a0a654ea16b1"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "8b59e8093e8fd289f1b1f30a48488c45"
server_key_iv: "e867212ef542122800087151fa5f647d"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.805346 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:10.834115 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:10.836679 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:10.836867 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:10.841004 19253 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34033
I20260924 15:30:10.841060 19322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34033 every 8 connection(s)
I20260924 15:30:10.841436 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:10.842435 19323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:10.843276 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19253
I20260924 15:30:10.843408 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:30:10.843600 5515 external_mini_cluster.cc:1428] Setting key a173c22314a5f8a3db9bd9206262a66f
I20260924 15:30:10.844990 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap starting.
I20260924 15:30:10.845762 19323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:10.846078 19323 log.cc:824] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:10.846829 19323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: No bootstrap required, opened a new log
I20260924 15:30:10.849172 19323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:10.849364 19323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:10.849408 19323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Initialized, Role: FOLLOWER
I20260924 15:30:10.849537 19323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:10.849640 19323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:10.849680 19323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:10.849731 19323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:10.850472 19323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:10.850565 19323 leader_election.cc:302] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06913431cd9d4f04bb26a0a654ea16b1; no voters:
I20260924 15:30:10.850762 19323 leader_election.cc:288] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:30:10.850782 19328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:10.850893 19328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 LEADER]: Becoming Leader. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Running, Role: LEADER
I20260924 15:30:10.850972 19328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:10.851018 19323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 15:30:10 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:10.851320 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:10.851457 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:10.851680 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06913431cd9d4f04bb26a0a654ea16b1. Latest consensus state: current_term: 1 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:10.851809 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:10.852106 19336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:10.852640 19336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:10.854250 19336 catalog_manager.cc:1347] Generated new cluster ID: 196d8d2942ac431f9fb57c201455709e
I20260924 15:30:10.854303 19336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:10.855397 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:10.844189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.5.98.254:34033)
0924 15:30:10.844378 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:10.844388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:10.844422 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:30:10.844870 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:10.844877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:10.844923 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:10.845043 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:10.845898 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:10.847142 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:10.848206 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:10.848640 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:10.848729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:10.851681 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:10.851686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:10.851686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:10.851698 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:10.853752 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:10.854153 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:10.854156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:10.854156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:10.854270 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:10.854507 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:10.854508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:10.854509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:10.854744 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:10.854910 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:10.855026 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:30:10.855180 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260924 15:30:10.863948 19336 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:30:10.864485 19336 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:10.867372 19336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Generated new TSK 0
I20260924 15:30:10.867540 19336 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:30:10.894166 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:10.975689 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:10.975875 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:10.975898 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:10.976945 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:10.976977 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:10.977002 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:10.978433 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:10.978605 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:10.978768 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:10.983609 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:10.983641 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:10.983631 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:10.983727 19349 server_base.cc:1034] running on GCE node
I20260924 15:30:10.983954 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:10.984194 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:10.985353 19349 hybrid_clock.cc:648] HybridClock initialized: now 1790263810985347 us; error 48 us; skew 500 ppm
Sep 24 15:30:10 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263810, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:10.987812 19349 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:10.988229 19349 webserver.cc:460] Webserver started at http://127.5.98.193:42915/ using document root <none> and password file <none>
I20260924 15:30:10.988468 19349 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:10.988546 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:10.988686 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:10.989706 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "5a4f579c339c428196963a775d031791"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "31d42271449cd249147d856a20140639"
server_key_iv: "c57b0c13c1fd2ace99edfb2ea1cc1695"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.990069 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4f579c339c428196963a775d031791"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "31d42271449cd249147d856a20140639"
server_key_iv: "c57b0c13c1fd2ace99edfb2ea1cc1695"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.991374 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:30:10.992231 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:10.992431 19349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:10.992513 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "5a4f579c339c428196963a775d031791"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "31d42271449cd249147d856a20140639"
server_key_iv: "c57b0c13c1fd2ace99edfb2ea1cc1695"
server_key_version: "encryptionkey@0"
I20260924 15:30:10.992566 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.013672 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.015960 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.016072 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.016253 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:11.016587 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:11.016624 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.016647 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:11.016696 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.022269 19349 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:42865
I20260924 15:30:11.022353 19479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:42865 every 8 connection(s)
I20260924 15:30:11.022668 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263810, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:11.028913 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19349
I20260924 15:30:11.029004 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:30:11.029225 5515 external_mini_cluster.cc:1428] Setting key 1bfe085b6eb6f8633e57af400a3e2c13
I20260924 15:30:11.032152 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.023425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:38107 (local address 127.5.98.254:34033)
0924 15:30:11.023488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.023490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.023710 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:30:11.024195 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.024196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.024206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.024256 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.025252 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.025746 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.026687 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.026918 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.026990 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.029144 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.029145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.029146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.029153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.030702 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.031131 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.031132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.031133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.031198 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.031460 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.031461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.031461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.031578 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.031648 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.031949 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.032062 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:30:11.032302 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.023395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:38107)
0924 15:30:11.023576 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.023619 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.024116 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.024318 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.024321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.024927 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.025140 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.025146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.025821 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.025822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.026611 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.026613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.027011 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.027012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.027159 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.027568 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.027576 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.029024 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.030801 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.030808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.030811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.031033 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.031274 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.031275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.031275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.031382 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.031653 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.031680 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.031816 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.031926 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.032078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260924 15:30:11.032827 19480 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:11.032922 19480 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:11.033113 19480 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:11.033591 19286 ts_manager.cc:194] Registered new tserver with Master: 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
I20260924 15:30:11.034211 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:38107
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:30:11.054203 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:11.134284 19485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:11.134474 19485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:11.134498 19485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:11.135581 19485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:11.135617 19485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:11.135649 19485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:11.137094 19485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:11.137269 19485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:11.137431 19485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:11.142256 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.142386 19485 server_base.cc:1034] running on GCE node
W20260924 15:30:11.142261 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.142274 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.142714 19485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:11.142956 19485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:11.144116 19485 hybrid_clock.cc:648] HybridClock initialized: now 1790263811144103 us; error 47 us; skew 500 ppm
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:11.146468 19485 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:11.146936 19485 webserver.cc:460] Webserver started at http://127.5.98.194:44741/ using document root <none> and password file <none>
I20260924 15:30:11.147156 19485 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:11.147224 19485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:11.147374 19485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:11.148442 19485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "64f30010af414ab3858c15e26506c960"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "5e84504c85eee465ff3111da3d2ad118"
server_key_iv: "3997f370881d99450ccd2aa8aca9bf80"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.148811 19485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "64f30010af414ab3858c15e26506c960"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "5e84504c85eee465ff3111da3d2ad118"
server_key_iv: "3997f370881d99450ccd2aa8aca9bf80"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.150113 19485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:30:11.150846 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.151120 19485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:11.151218 19485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "64f30010af414ab3858c15e26506c960"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "5e84504c85eee465ff3111da3d2ad118"
server_key_iv: "3997f370881d99450ccd2aa8aca9bf80"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.151302 19485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.166997 19485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.168886 19485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.169040 19485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.169248 19485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:11.169556 19485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:11.169610 19485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.169649 19485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:11.169677 19485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.175565 19485 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46233
I20260924 15:30:11.175602 19615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46233 every 8 connection(s)
I20260924 15:30:11.175925 19485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:30:11.178483 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19485
I20260924 15:30:11.178586 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:30:11.178810 5515 external_mini_cluster.cc:1428] Setting key 74ae7a66afc4ce4fd51b3bf01700fb32
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:11.185981 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.176630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:57845 (local address 127.5.98.254:34033)
0924 15:30:11.176694 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.176696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.176885 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:30:11.177419 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.177421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.177434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.177481 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.178423 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.179190 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.180385 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.180602 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.180677 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.183132 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.183135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.183136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.183147 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.184565 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.185055 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.185055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.185056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.185103 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.185381 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.185381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.185382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.185476 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.185524 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.185856 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.185921 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:30:11.186245 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.176599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:57845)
0924 15:30:11.176752 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.176797 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.177300 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.177548 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.177552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.178114 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.178318 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.178322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.179281 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.179282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.180284 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.180287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.180684 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.180685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.180819 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.181333 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.181341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.183025 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.184648 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.184655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.184658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.184973 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.185174 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.185175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.185176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.185314 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.185541 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.185547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.185711 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.185839 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.186009 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260924 15:30:11.186769 19616 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:11.186863 19616 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:11.187048 19616 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:11.187431 19286 ts_manager.cc:194] Registered new tserver with Master: 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233)
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:30:11.187840 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:57845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:30:11.204370 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:11.284112 19621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:11.284333 19621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:11.284368 19621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:11.285478 19621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:11.285540 19621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:11.285583 19621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:11.286983 19621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:11.287236 19621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:11.287451 19621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:11.292429 19629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.292483 19626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.292465 19627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.292516 19621 server_base.cc:1034] running on GCE node
I20260924 15:30:11.292805 19621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:11.293038 19621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:11.294195 19621 hybrid_clock.cc:648] HybridClock initialized: now 1790263811294191 us; error 37 us; skew 500 ppm
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:11.296684 19621 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:11.297133 19621 webserver.cc:460] Webserver started at http://127.5.98.195:37165/ using document root <none> and password file <none>
I20260924 15:30:11.297353 19621 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:11.297418 19621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:11.297567 19621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:11.298571 19621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "b4fa8e25bc554cf18356f839ad85cd54"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "0025ca26dfb24ad1196c1bfa591bdac6"
server_key_iv: "fe122a0f9c4764e44ae683476a1e9a62"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.298925 19621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "b4fa8e25bc554cf18356f839ad85cd54"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "0025ca26dfb24ad1196c1bfa591bdac6"
server_key_iv: "fe122a0f9c4764e44ae683476a1e9a62"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.300280 19621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:30:11.300962 19636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.301131 19621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:11.301226 19621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "b4fa8e25bc554cf18356f839ad85cd54"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "0025ca26dfb24ad1196c1bfa591bdac6"
server_key_iv: "fe122a0f9c4764e44ae683476a1e9a62"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.301307 19621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.328204 19621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.329995 19621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.330152 19621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.330384 19621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:11.330678 19621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:11.330729 19621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.330770 19621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:11.330797 19621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.336256 19621 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46347
I20260924 15:30:11.336313 19749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46347 every 8 connection(s)
I20260924 15:30:11.336629 19621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:11.338855 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19621
I20260924 15:30:11.338963 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:30:11.339161 5515 external_mini_cluster.cc:1428] Setting key 2a0fe00cf59860fb334631d07331f0ec
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:11.345618 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.337307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:43713 (local address 127.5.98.254:34033)
0924 15:30:11.337390 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.337393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.337575 (+ 182us) server_negotiation.cc:407] Connection header received
0924 15:30:11.338107 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.338108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.338122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.338174 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.339332 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.340039 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.341032 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.341217 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.341273 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.343290 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.343291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.343291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.343296 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.344488 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.344841 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.344841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.344842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.344894 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.345105 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.345106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.345106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.345200 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.345248 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.345489 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.345552 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:30:11.345834 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.337280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:43713)
0924 15:30:11.337453 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.337487 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.337988 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.338248 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.338252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.338933 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.339227 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.339235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.340136 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.340137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.340949 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.340951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.341291 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.341291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.341387 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.341910 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.341918 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.343216 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.344564 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.344571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.344572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.344774 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.344955 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.344955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.344956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.345050 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.345255 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.345259 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.345368 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.345453 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.345587 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260924 15:30:11.346357 19750 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:11.346453 19750 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:11.346623 19750 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:11.347005 19286 ts_manager.cc:194] Registered new tserver with Master: b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347)
I20260924 15:30:11.347384 19286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:43713
I20260924 15:30:11.350812 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:11.358161 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.352193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.5.98.254:34033)
0924 15:30:11.352227 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.352228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.352364 (+ 136us) server_negotiation.cc:407] Connection header received
0924 15:30:11.352479 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.352480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.352490 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.352545 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.353092 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.353560 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.354355 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.354567 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.354639 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.355772 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.355773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.355774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.355780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.356935 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.357305 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.357306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.357307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.357390 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.357596 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.357597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.357598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.357735 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.357844 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.357998 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.358069 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:30:11.359460 19286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:30:11.361061 19286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:30:11.367228 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.0.0.1:41256)
0924 15:30:11.363604 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.363615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.363712 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.363998 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.363999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.364002 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:11.364047 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.364049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.364980 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.364981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.366201 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.366204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.366931 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.366932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.367011 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.367027 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.367098 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.367157 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":45,"threads_started":1}
I20260924 15:30:11.367362 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.0.0.1:42514)
0924 15:30:11.363507 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.363548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.363676 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.363930 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.363933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.363935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:11.363988 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.363990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.365334 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.365335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.366384 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.366386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.367108 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.367109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.367174 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.367178 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.367234 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.367275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
I20260924 15:30:11.367264 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.5.98.193:42865)
0924 15:30:11.363604 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.363608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.363683 (+ 75us) server_negotiation.cc:407] Connection header received
0924 15:30:11.363819 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.363822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.363839 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.363914 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:11.364128 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.364871 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.366288 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.366847 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.366905 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.366944 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.367125 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.367166 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260924 15:30:11.367624 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.5.98.194:46233)
0924 15:30:11.363481 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.363485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.363615 (+ 130us) server_negotiation.cc:407] Connection header received
0924 15:30:11.363768 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.363770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.363794 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.363872 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:11.364086 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.365228 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.366466 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.367000 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.367079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.367136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.367462 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.367518 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260924 15:30:11.368330 19412 tablet_service.cc:1467] Processing CreateTablet for tablet e610e209dab84c5a8db6b00250caf9f4 (DEFAULT_TABLE table=test-table [id=0bdfeda123264df3886bd01bb68b1c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:11.368453 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46347 (local address 127.0.0.1:43868)
0924 15:30:11.363807 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.363818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.363857 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.364181 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.364182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.364185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:11.364222 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.364223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.365440 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.365441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.366378 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.366380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.368231 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.368232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.368294 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.368298 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.368346 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.368383 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":47,"threads_started":1}
I20260924 15:30:11.368531 19548 tablet_service.cc:1467] Processing CreateTablet for tablet e610e209dab84c5a8db6b00250caf9f4 (DEFAULT_TABLE table=test-table [id=0bdfeda123264df3886bd01bb68b1c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:11.368716 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e610e209dab84c5a8db6b00250caf9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:11.368727 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.363603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.5.98.195:46347)
0924 15:30:11.363973 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.363981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.363991 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:11.364025 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.364027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.364042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.364112 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:11.364305 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.365331 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.367193 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.368073 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.368144 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.368197 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.368610 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.368650 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260924 15:30:11.368889 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e610e209dab84c5a8db6b00250caf9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:11.369293 19684 tablet_service.cc:1467] Processing CreateTablet for tablet e610e209dab84c5a8db6b00250caf9f4 (DEFAULT_TABLE table=test-table [id=0bdfeda123264df3886bd01bb68b1c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:11.369593 19684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e610e209dab84c5a8db6b00250caf9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:11.371394 19764 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Bootstrap starting.
I20260924 15:30:11.372083 19764 tablet_bootstrap.cc:654] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:11.372623 19766 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Bootstrap starting.
I20260924 15:30:11.372673 19764 log.cc:824] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:11.373027 19767 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Bootstrap starting.
I20260924 15:30:11.373349 19766 tablet_bootstrap.cc:654] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:11.373605 19767 tablet_bootstrap.cc:654] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:11.373836 19766 log.cc:824] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:11.374125 19767 log.cc:824] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:11.374655 19766 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: No bootstrap required, opened a new log
I20260924 15:30:11.374761 19766 ts_tablet_manager.cc:1397] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:11.374919 19764 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: No bootstrap required, opened a new log
I20260924 15:30:11.374929 19767 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: No bootstrap required, opened a new log
I20260924 15:30:11.374982 19767 ts_tablet_manager.cc:1397] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:11.375002 19764 ts_tablet_manager.cc:1397] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 15:30:11.376403 19764 raft_consensus.cc:348] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.376569 19764 raft_consensus.cc:374] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:11.376614 19764 raft_consensus.cc:729] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4f579c339c428196963a775d031791, State: Initialized, Role: FOLLOWER
I20260924 15:30:11.376721 19764 consensus_queue.cc:260] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.376802 19766 raft_consensus.cc:348] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.376853 19767 raft_consensus.cc:348] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.376981 19766 raft_consensus.cc:374] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:11.376996 19767 raft_consensus.cc:374] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:11.377007 19764 ts_tablet_manager.cc:1428] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:11.377027 19766 raft_consensus.cc:729] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f30010af414ab3858c15e26506c960, State: Initialized, Role: FOLLOWER
I20260924 15:30:11.377029 19767 raft_consensus.cc:729] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fa8e25bc554cf18356f839ad85cd54, State: Initialized, Role: FOLLOWER
I20260924 15:30:11.377152 19480 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
I20260924 15:30:11.377156 19767 consensus_queue.cc:260] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.377432 19767 ts_tablet_manager.cc:1428] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:30:11.377450 19750 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
I20260924 15:30:11.377444 19766 consensus_queue.cc:260] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.377707 19766 ts_tablet_manager.cc:1428] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:30:11.377722 19616 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
W20260924 15:30:11.426602 19617 tablet.cc:2369] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:30:11.460856 19772 raft_consensus.cc:482] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:11.461011 19772 raft_consensus.cc:504] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.461409 19772 leader_election.cc:288] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233), 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
I20260924 15:30:11.464527 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.461684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:47429 (local address 127.5.98.194:46233)
0924 15:30:11.461733 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.461736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.461747 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:30:11.461832 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.461833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.461845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.461907 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:11.462119 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.462638 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.463739 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.464131 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.464189 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.464196 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.464398 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.464440 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:30:11.464524 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.461608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.5.98.195:47429)
0924 15:30:11.461675 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.461688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.461745 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.461943 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.461945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.461947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:11.462036 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.462038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.462721 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.462722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.463639 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.463642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.464213 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.464214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.464312 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.464327 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.464387 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.464435 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260924 15:30:11.464646 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.461608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.5.98.195:52479)
0924 15:30:11.461840 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.461853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.461911 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.462064 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.462065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.462067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:11.462168 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.462170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.462908 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.462909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.463642 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.463643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.464356 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.464357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.464444 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.464449 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.464508 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.464554 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":24,"thread_start_us":147,"threads_started":1}
I20260924 15:30:11.464686 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.461693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:52479 (local address 127.5.98.193:42865)
0924 15:30:11.461736 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.461738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.461930 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:30:11.461965 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.461967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.461981 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.462048 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:11.462250 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.462802 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.463783 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.464252 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.464342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.464352 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.464533 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.464581 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:30:11.464895 19569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e610e209dab84c5a8db6b00250caf9f4" candidate_uuid: "b4fa8e25bc554cf18356f839ad85cd54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f30010af414ab3858c15e26506c960" is_pre_election: true
I20260924 15:30:11.465070 19569 raft_consensus.cc:2454] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fa8e25bc554cf18356f839ad85cd54 in term 0.
I20260924 15:30:11.465065 19433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e610e209dab84c5a8db6b00250caf9f4" candidate_uuid: "b4fa8e25bc554cf18356f839ad85cd54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4f579c339c428196963a775d031791" is_pre_election: true
I20260924 15:30:11.465298 19433 raft_consensus.cc:2454] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fa8e25bc554cf18356f839ad85cd54 in term 0.
I20260924 15:30:11.465346 19640 leader_election.cc:302] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f30010af414ab3858c15e26506c960, b4fa8e25bc554cf18356f839ad85cd54; no voters:
I20260924 15:30:11.465471 19772 raft_consensus.cc:2781] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:11.465556 19772 raft_consensus.cc:482] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:11.465583 19772 raft_consensus.cc:3037] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:11.466305 19772 raft_consensus.cc:504] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.466419 19772 leader_election.cc:288] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [CANDIDATE]: Term 1 election: Requested vote from peers 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233), 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
I20260924 15:30:11.466662 19569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e610e209dab84c5a8db6b00250caf9f4" candidate_uuid: "b4fa8e25bc554cf18356f839ad85cd54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f30010af414ab3858c15e26506c960"
I20260924 15:30:11.466662 19433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e610e209dab84c5a8db6b00250caf9f4" candidate_uuid: "b4fa8e25bc554cf18356f839ad85cd54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4f579c339c428196963a775d031791"
I20260924 15:30:11.466745 19433 raft_consensus.cc:3037] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:11.466745 19569 raft_consensus.cc:3037] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:11.467633 19569 raft_consensus.cc:2454] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fa8e25bc554cf18356f839ad85cd54 in term 1.
I20260924 15:30:11.467633 19433 raft_consensus.cc:2454] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fa8e25bc554cf18356f839ad85cd54 in term 1.
I20260924 15:30:11.467839 19638 leader_election.cc:302] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4f579c339c428196963a775d031791, b4fa8e25bc554cf18356f839ad85cd54; no voters:
I20260924 15:30:11.467953 19772 raft_consensus.cc:2781] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:11.468155 19772 raft_consensus.cc:686] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 LEADER]: Becoming Leader. State: Replica: b4fa8e25bc554cf18356f839ad85cd54, State: Running, Role: LEADER
I20260924 15:30:11.468271 19772 consensus_queue.cc:237] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.469195 19285 catalog_manager.cc:5495] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "b4fa8e25bc554cf18356f839ad85cd54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } health_report { overall_health: HEALTHY } } }
I20260924 15:30:11.488827 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.486094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.5.98.195:46347)
0924 15:30:11.486149 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.486151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.486160 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:30:11.486208 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.486209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.486231 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.486331 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:11.486474 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.487008 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.487896 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.488137 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.488225 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.488411 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:11.488538 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:11.488645 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.488673 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.488695 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:30:11.491158 19569 raft_consensus.cc:1261] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Refusing update from remote peer b4fa8e25bc554cf18356f839ad85cd54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:11.491226 19433 raft_consensus.cc:1261] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Refusing update from remote peer b4fa8e25bc554cf18356f839ad85cd54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:11.491676 19772 consensus_queue.cc:1035] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:11.491827 19775 consensus_queue.cc:1035] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:11.494899 19778 mvcc.cc:205] Tried to move back new op lower bound from 7332920571864784896 to 7332920571774865408. Current Snapshot: MvccSnapshot[applied={T|T < 7332920571864784896}]
I20260924 15:30:11.495649 19780 mvcc.cc:205] Tried to move back new op lower bound from 7332920571864784896 to 7332920571774865408. Current Snapshot: MvccSnapshot[applied={T|T < 7332920571864784896 or (T in {7332920571864784896})}]
I20260924 15:30:11.496411 19779 mvcc.cc:205] Tried to move back new op lower bound from 7332920571864784896 to 7332920571774865408. Current Snapshot: MvccSnapshot[applied={T|T < 7332920571864784896 or (T in {7332920571864784896})}]
I20260924 15:30:11.498142 19285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:11.498242 19285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:11.499169 19285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Sending DeleteTablet for 3 replicas of tablet e610e209dab84c5a8db6b00250caf9f4
I20260924 15:30:11.499570 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet e610e209dab84c5a8db6b00250caf9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:11 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:41256
I20260924 15:30:11.499593 19684 tablet_service.cc:1514] Processing DeleteTablet for tablet e610e209dab84c5a8db6b00250caf9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:11 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:43868
I20260924 15:30:11.499879 19787 tablet_replica.cc:330] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: stopping tablet replica
I20260924 15:30:11.499898 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet e610e209dab84c5a8db6b00250caf9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:11 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42514
I20260924 15:30:11.500016 19787 raft_consensus.cc:2229] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:30:11.500155 19789 tablet_replica.cc:330] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: stopping tablet replica
I20260924 15:30:11.500180 19787 raft_consensus.cc:2258] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:11.500305 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19349
I20260924 15:30:11.500329 19789 raft_consensus.cc:2229] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:11.500527 19789 raft_consensus.cc:2258] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:11.501432 19789 ts_tablet_manager.cc:1905] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:11.501719 19787 ts_tablet_manager.cc:1905] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:11.503207 19789 ts_tablet_manager.cc:1918] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:11.503360 19789 log.cc:1196] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/e610e209dab84c5a8db6b00250caf9f4
I20260924 15:30:11.503775 19789 ts_tablet_manager.cc:1939] T e610e209dab84c5a8db6b00250caf9f4 P 64f30010af414ab3858c15e26506c960: Deleting consensus metadata
I20260924 15:30:11.503912 19787 ts_tablet_manager.cc:1918] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:11.503994 19787 log.cc:1196] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/e610e209dab84c5a8db6b00250caf9f4
I20260924 15:30:11.504282 19787 ts_tablet_manager.cc:1939] T e610e209dab84c5a8db6b00250caf9f4 P b4fa8e25bc554cf18356f839ad85cd54: Deleting consensus metadata
I20260924 15:30:11.504427 19273 catalog_manager.cc:4841] TS 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233): tablet e610e209dab84c5a8db6b00250caf9f4 (table test-table [id=0bdfeda123264df3886bd01bb68b1c71]) successfully deleted
I20260924 15:30:11.504664 19273 catalog_manager.cc:4841] TS b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347): tablet e610e209dab84c5a8db6b00250caf9f4 (table test-table [id=0bdfeda123264df3886bd01bb68b1c71]) successfully deleted
W20260924 15:30:11.507792 19271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:42865 (error 108)
I20260924 15:30:11.508020 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19485
I20260924 15:30:11.508673 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.508135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.0.0.1:41270)
0924 15:30:11.508577 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.508626 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:42865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260924 15:30:11.509456 19271 catalog_manager.cc:4584] TS 5a4f579c339c428196963a775d031791 (127.5.98.193:42865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e610e209dab84c5a8db6b00250caf9f4: Network error: Client connection negotiation failed: client connection to 127.5.98.193:42865: connect: Connection refused (error 111)
I20260924 15:30:11.518641 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19621
I20260924 15:30:11.525700 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19253
I20260924 15:30:11.533087 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--webserver_interface=127.5.98.254
--webserver_port=40701
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:11.622197 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:11.622448 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:11.622485 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:11.623667 19791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:11.623754 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:11.623777 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:11.623797 19791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:11.623826 19791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:11.625270 19791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:11.625545 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:11.625766 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:11.634611 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.634606 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.634606 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.634872 19791 server_base.cc:1034] running on GCE node
I20260924 15:30:11.635098 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:11.635357 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:11.636503 19791 hybrid_clock.cc:648] HybridClock initialized: now 1790263811636488 us; error 28 us; skew 500 ppm
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:11.645557 19791 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:11.646064 19791 webserver.cc:460] Webserver started at http://127.5.98.254:40701/ using document root <none> and password file <none>
I20260924 15:30:11.646307 19791 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:11.646391 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:11.647794 19791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:11.648497 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.648670 19791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:11.648780 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "06913431cd9d4f04bb26a0a654ea16b1"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "8b59e8093e8fd289f1b1f30a48488c45"
server_key_iv: "e867212ef542122800087151fa5f647d"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.649185 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.658718 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.660514 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.660668 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.664649 19791 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34033
I20260924 15:30:11.664677 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34033 every 8 connection(s)
I20260924 15:30:11.665055 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:11.666649 19861 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:30:11.667246 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap starting.
I20260924 15:30:11.668562 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19791
I20260924 15:30:11.668836 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:42865
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=42915
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:11.675626 19861 log.cc:824] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:11.677635 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:11.677947 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap complete.
I20260924 15:30:11.680305 19861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:11.680563 19861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Initialized, Role: FOLLOWER
I20260924 15:30:11.680750 19861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:11.680830 19861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:11.680886 19861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:11.680938 19861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:30:11.681923 19861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:11.682045 19861 leader_election.cc:302] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06913431cd9d4f04bb26a0a654ea16b1; no voters:
I20260924 15:30:11.682232 19861 leader_election.cc:288] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:30:11.682283 19865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:30:11.682472 19865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 LEADER]: Becoming Leader. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Running, Role: LEADER
I20260924 15:30:11.682511 19861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:11.682600 19865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:11.683058 19867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:11.683172 19867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:11.683063 19865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06913431cd9d4f04bb26a0a654ea16b1. Latest consensus state: current_term: 2 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:11.683365 19865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:11.683951 19875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:11.684855 19875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:11.684958 19875 catalog_manager.cc:1259] Loaded cluster ID: 196d8d2942ac431f9fb57c201455709e
I20260924 15:30:11.684993 19875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:11.685269 19875 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 15:30:11.689918 19875 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 15:30:11.692312 19875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:11.692497 19875 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Loaded TSK: 0
I20260924 15:30:11.692734 19875 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:30:11.755101 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:11.755323 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:11.755362 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:11.756554 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:11.756620 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:11.756665 19863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:11.758116 19863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:42865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=42915
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:11.758368 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:11.758591 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:11.764459 19889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.764504 19890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.764539 19892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.764823 19863 server_base.cc:1034] running on GCE node
I20260924 15:30:11.765024 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:11.765264 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:11.766425 19863 hybrid_clock.cc:648] HybridClock initialized: now 1790263811766384 us; error 61 us; skew 500 ppm
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:11.769083 19863 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:11.769505 19863 webserver.cc:460] Webserver started at http://127.5.98.193:42915/ using document root <none> and password file <none>
I20260924 15:30:11.769744 19863 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:11.769824 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:11.771246 19863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:11.771994 19899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.772194 19863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:11.772298 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "5a4f579c339c428196963a775d031791"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "31d42271449cd249147d856a20140639"
server_key_iv: "c57b0c13c1fd2ace99edfb2ea1cc1695"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.772650 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.788376 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.790498 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.790659 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.790884 19863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:11.791319 19906 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:30:11.793285 19863 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:30:11.793342 19863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:30:11.793396 19863 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:30:11.794040 19863 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:30:11.794094 19863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:11.794162 19906 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Bootstrap starting.
I20260924 15:30:11.801329 19863 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:42865
I20260924 15:30:11.801393 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:42865 every 8 connection(s)
I20260924 15:30:11.801762 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:30:11.804286 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 19863
I20260924 15:30:11.804584 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:46233
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=44741
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:11.804962 19906 log.cc:824] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:11.807512 19906 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:11.807842 19906 tablet_bootstrap.cc:492] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Bootstrap complete.
I20260924 15:30:11.809464 19906 ts_tablet_manager.cc:1397] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.007s
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:11.811806 19906 raft_consensus.cc:348] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.812507 19906 raft_consensus.cc:729] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4f579c339c428196963a775d031791, State: Initialized, Role: FOLLOWER
I20260924 15:30:11.812724 19906 consensus_queue.cc:260] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } }
I20260924 15:30:11.813156 19906 ts_tablet_manager.cc:1428] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 15:30:11.815770 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.802455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:58893)
0924 15:30:11.802721 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.802752 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.803160 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.803796 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.803800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.804112 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.804417 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.804423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.806148 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.806149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.807476 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.807480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.808427 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.808428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.808578 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.809197 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.809206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.811082 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.813640 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.813647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.813650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.813990 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.814308 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.814309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.814309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.814445 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.814935 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.814942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.815133 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.815244 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.815482 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260924 15:30:11.816270 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.802511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:58893 (local address 127.5.98.254:34033)
0924 15:30:11.803396 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.803403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.803428 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:30:11.803537 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.803541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.803580 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.803766 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.804562 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.806015 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.807587 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.808300 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.808872 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.811211 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.811215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.811215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.811229 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.813530 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.814097 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.814100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.814101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.814220 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.814529 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.814530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.814530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.814839 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.815842 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.815919 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.816068 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":79,"threads_started":1}
I20260924 15:30:11.816931 20015 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:11.817065 20015 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:11.817260 20015 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:11.818104 19825 ts_manager.cc:194] Registered new tserver with Master: 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
I20260924 15:30:11.819273 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:58893
I20260924 15:30:11.820258 20015 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:30:11.829107 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.818696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.0.0.1:41278)
0924 15:30:11.819615 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.819698 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.819776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.820044 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.820050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.820447 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.820748 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.820751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.821897 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.821898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.822887 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.822889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.823505 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.823507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.823592 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.824118 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.824124 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.825701 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.827848 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.827855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.827857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.828110 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.828347 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.828348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.828348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.828414 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.828729 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.828735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.828790 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.828880 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.829016 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":83,"threads_started":1}
I20260924 15:30:11.829107 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.818939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.5.98.193:42865)
0924 15:30:11.819095 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.819099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.819856 (+ 757us) server_negotiation.cc:407] Connection header received
0924 15:30:11.819898 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.819902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.819921 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.819993 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.820839 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.821810 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.822974 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.823417 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.823485 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.825792 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.825800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.825801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.825808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.827751 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.828196 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.828199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.828199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.828260 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.828489 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.828491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.828491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.828655 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.828731 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.828927 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.829008 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260924 15:30:11.830291 19949 tablet_service.cc:1514] Processing DeleteTablet for tablet e610e209dab84c5a8db6b00250caf9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:11 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:41278
I20260924 15:30:11.830761 20024 tablet_replica.cc:330] stopping tablet replica
I20260924 15:30:11.830891 20024 raft_consensus.cc:2229] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:11.830971 20024 raft_consensus.cc:2258] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:11.831305 20024 ts_tablet_manager.cc:1905] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:11.833226 20024 ts_tablet_manager.cc:1918] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:11.833289 20024 log.cc:1196] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/e610e209dab84c5a8db6b00250caf9f4
I20260924 15:30:11.833506 20024 ts_tablet_manager.cc:1939] T e610e209dab84c5a8db6b00250caf9f4 P 5a4f579c339c428196963a775d031791: Deleting consensus metadata
I20260924 15:30:11.833967 19810 catalog_manager.cc:4841] TS 5a4f579c339c428196963a775d031791 (127.5.98.193:42865): tablet e610e209dab84c5a8db6b00250caf9f4 (table test-table [id=0bdfeda123264df3886bd01bb68b1c71]) successfully deleted
W20260924 15:30:11.892792 20020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:11.893024 20020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:11.893059 20020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:11.894191 20020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:11.894255 20020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:11.894301 20020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:11.895828 20020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:46233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=44741
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:11.896086 20020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:11.896319 20020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:11.901253 20029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:11.901261 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.901372 20020 server_base.cc:1034] running on GCE node
W20260924 15:30:11.901261 20030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:11.901727 20020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:11.901925 20020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:11.903084 20020 hybrid_clock.cc:648] HybridClock initialized: now 1790263811903081 us; error 34 us; skew 500 ppm
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:11.905560 20020 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:11.905977 20020 webserver.cc:460] Webserver started at http://127.5.98.194:44741/ using document root <none> and password file <none>
I20260924 15:30:11.906206 20020 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:11.906288 20020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:11.907596 20020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:11.908179 20040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.908344 20020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.908448 20020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "64f30010af414ab3858c15e26506c960"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "5e84504c85eee465ff3111da3d2ad118"
server_key_iv: "3997f370881d99450ccd2aa8aca9bf80"
server_key_version: "encryptionkey@0"
I20260924 15:30:11.908784 20020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:11.932087 20020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:11.933950 20020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:11.934129 20020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:11.934376 20020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:11.934696 20020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:11.934752 20020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.934798 20020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:11.934820 20020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:11.940518 20020 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46233
I20260924 15:30:11.940577 20156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46233 every 8 connection(s)
I20260924 15:30:11.940941 20020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:30:11 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:11.949600 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.941727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:56733 (local address 127.5.98.254:34033)
0924 15:30:11.941774 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:11.941776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:11.941985 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:30:11.942443 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:11.942444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:11.942456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:11.942508 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:11.943423 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.943980 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.944915 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.945114 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.945164 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.947186 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:11.947187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:11.947188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:11.947194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:11.948465 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.948822 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.948823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.948823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.948871 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:11.949086 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:11.949086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:11.949087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:11.949184 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:11.949234 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:11.949454 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 15:30:11.949521 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:30:11.949802 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:11.941711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:56733)
0924 15:30:11.941860 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:11.941895 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:11.942360 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:11.942564 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:11.942567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:11.943147 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:11.943347 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.943352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.944057 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.944058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.944841 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:11.944843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:11.945185 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:11.945186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:11.945283 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:11.945799 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:11.945807 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:11.947100 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:11.948542 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.948549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.948551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.948757 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.948928 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:11.948929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:11.948929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:11.949032 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:11.949237 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:11.949241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:11.949350 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:30:11.949427 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:30:11.949546 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260924 15:30:11.950424 20157 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:11.950551 20157 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:11.950618 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20020
I20260924 15:30:11.950765 20157 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:11.950898 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:46347
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=37165
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:11.951172 19825 ts_manager.cc:194] Registered new tserver with Master: 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233)
I20260924 15:30:11.951619 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:56733
W20260924 15:30:12.032881 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:12.033106 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:12.033155 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:12.034314 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:12.034379 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:12.034425 20160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:12.035962 20160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:46347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=37165
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:12.036223 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:12.036451 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:12.041404 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.041494 20169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.041415 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.041730 20160 server_base.cc:1034] running on GCE node
I20260924 15:30:12.041927 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:12.042137 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:12.043303 20160 hybrid_clock.cc:648] HybridClock initialized: now 1790263812043286 us; error 32 us; skew 500 ppm
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:12.045682 20160 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:12.046150 20160 webserver.cc:460] Webserver started at http://127.5.98.195:37165/ using document root <none> and password file <none>
I20260924 15:30:12.046396 20160 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:12.046479 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:12.047876 20160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:12.048430 20176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.048596 20160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.048699 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "b4fa8e25bc554cf18356f839ad85cd54"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "0025ca26dfb24ad1196c1bfa591bdac6"
server_key_iv: "fe122a0f9c4764e44ae683476a1e9a62"
server_key_version: "encryptionkey@0"
I20260924 15:30:12.049041 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:12.073516 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:12.075415 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:12.075584 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:12.075860 20160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:12.076220 20160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:12.076272 20160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.076306 20160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:12.076333 20160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.082072 20160 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46347
I20260924 15:30:12.082120 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46347 every 8 connection(s)
I20260924 15:30:12.082453 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:12.086335 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20160
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:12.092326 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.083267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:57831 (local address 127.5.98.254:34033)
0924 15:30:12.083369 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.083389 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.083580 (+ 191us) server_negotiation.cc:407] Connection header received
0924 15:30:12.084210 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.084212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.084224 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.084277 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:12.085225 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.085728 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.086992 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.087203 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.087283 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.089396 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:12.089398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:12.089399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:12.089407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:12.090889 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:12.091270 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:12.091271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:12.091272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:12.091328 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:12.091574 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:12.091575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:12.091576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:12.091760 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:12.091869 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.092149 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.092236 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 15:30:12.092506 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.083278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:57831)
0924 15:30:12.083460 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.083498 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.084103 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.084361 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:12.084365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:12.084917 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:12.085122 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.085126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.085826 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.085827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.086894 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.086898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.087288 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.087289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.087415 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.087993 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:12.088001 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:12.089298 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:12.090968 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:12.090974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:12.090976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:12.091191 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:12.091402 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:12.091402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:12.091403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:12.091495 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:12.091839 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:12.091846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:12.091994 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 15:30:12.092114 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 15:30:12.092271 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260924 15:30:12.093003 20292 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34033
I20260924 15:30:12.093092 20292 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:12.093314 20292 heartbeater.cc:511] Master 127.5.98.254:34033 requested a full tablet report, sending...
I20260924 15:30:12.093761 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.086721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.5.98.254:34033)
0924 15:30:12.086837 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.086839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.086858 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:30:12.086913 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.086914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.086926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.086991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:12.087540 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.088100 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.088836 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.089128 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.089204 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.090579 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:12.090580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:12.090580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:12.090587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:12.092168 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:12.092771 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:12.092772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:12.092773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:12.092835 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:12.093181 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:12.093182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:12.093182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:12.093307 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:12.093494 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.093589 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.093651 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260924 15:30:12.093765 19825 ts_manager.cc:194] Registered new tserver with Master: b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347)
I20260924 15:30:12.094233 19825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:57831
I20260924 15:30:12.094470 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:12.096982 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.094840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.5.98.254:34033)
0924 15:30:12.094876 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.094878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.094944 (+ 66us) server_negotiation.cc:407] Connection header received
0924 15:30:12.095018 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.095019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.095044 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.095109 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:12.095221 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.095639 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.096360 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.096539 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.096588 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.096717 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:12.096797 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:12.096849 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.096880 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.096895 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:30:12.097252 19825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:30:12.098935 19825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:30:12.100996 19949 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 (DEFAULT_TABLE table=test-table [id=5f180bcfd39f48818f5b8bb1674459ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:12.101184 19949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:12.102504 19906 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Bootstrap starting.
I20260924 15:30:12.102804 19906 tablet_bootstrap.cc:654] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:12.103425 19906 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: No bootstrap required, opened a new log
I20260924 15:30:12.103477 19906 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:12.103827 19906 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.103920 19906 raft_consensus.cc:374] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:12.103971 19906 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4f579c339c428196963a775d031791, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.104028 19906 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.104143 19906 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:12.105118 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.100826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46347 (local address 127.0.0.1:43880)
0924 15:30:12.100956 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.100967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.101020 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.101546 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:12.101547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:12.101548 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:12.101582 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.101583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.102858 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.102859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.103971 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.103974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.104840 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.104840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.104915 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.104934 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 15:30:12.104988 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:30:12.105034 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260924 15:30:12.105119 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.100744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.0.0.1:42520)
0924 15:30:12.100944 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.100955 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.101002 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.101258 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:12.101259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:12.101261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:12.101298 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.101299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.102625 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.102626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.103773 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.103776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.104907 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.104908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.104961 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.104964 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:30:12.105021 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:30:12.105056 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260924 15:30:12.105142 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.100869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.5.98.194:46233)
0924 15:30:12.101071 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.101075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.101085 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:12.101129 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.101131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.101147 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.101236 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:12.101375 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.102503 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.103971 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.104838 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.104893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.104927 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.105033 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.105068 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260924 15:30:12.105145 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.101136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.5.98.195:46347)
0924 15:30:12.101392 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.101397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.101407 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:12.101441 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.101443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.101458 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.101534 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:12.101644 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.102758 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.104081 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.104685 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.104733 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.104790 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.105029 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.105068 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":69,"threads_started":1}
I20260924 15:30:12.106012 20090 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 (DEFAULT_TABLE table=test-table [id=5f180bcfd39f48818f5b8bb1674459ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:12.106269 20225 tablet_service.cc:1467] Processing CreateTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 (DEFAULT_TABLE table=test-table [id=5f180bcfd39f48818f5b8bb1674459ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:12.106365 20090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:12.106580 20225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:12.109174 20302 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Bootstrap starting.
I20260924 15:30:12.109400 20303 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Bootstrap starting.
I20260924 15:30:12.109781 20302 tablet_bootstrap.cc:654] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:12.109959 20303 tablet_bootstrap.cc:654] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:12.110277 20302 log.cc:824] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:12.110435 20303 log.cc:824] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:12.111002 20302 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: No bootstrap required, opened a new log
I20260924 15:30:12.111086 20302 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:12.111171 20303 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: No bootstrap required, opened a new log
I20260924 15:30:12.111244 20303 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:12.112659 20302 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.112789 20302 raft_consensus.cc:374] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:12.112811 20302 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f30010af414ab3858c15e26506c960, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.112749 20303 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.112934 20303 raft_consensus.cc:374] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:12.112918 20302 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.112986 20303 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fa8e25bc554cf18356f839ad85cd54, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.113091 20303 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.113197 20302 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:12.113294 20157 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
I20260924 15:30:12.113402 20303 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:12.113423 20292 heartbeater.cc:503] Master 127.5.98.254:34033 was elected leader, sending a full tablet report...
W20260924 15:30:12.191565 20158 tablet.cc:2369] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:30:12.303694 20016 tablet.cc:2369] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:30:12.333091 20293 tablet.cc:2369] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:30:12.334450 20021 raft_consensus.cc:482] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:12.334559 20021 raft_consensus.cc:504] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.334837 20021 leader_election.cc:288] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233), b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347)
I20260924 15:30:12.338018 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.335345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:46813 (local address 127.5.98.195:46347)
0924 15:30:12.335405 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.335407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.335450 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:30:12.335546 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.335546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.335556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.335630 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:12.335791 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.336264 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337254 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.337638 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.337735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.337911 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.337951 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:30:12.338042 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.335262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46347 (local address 127.5.98.193:46813)
0924 15:30:12.335388 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.335399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.335469 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.335634 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:12.335635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:12.335636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:12.335714 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.335716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.336354 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.336355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.337152 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.337154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.337727 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.337793 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.337823 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 15:30:12.337901 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:30:12.337947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":31,"thread_start_us":43,"threads_started":1}
I20260924 15:30:12.338109 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.335150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:49919 (local address 127.5.98.194:46233)
0924 15:30:12.335218 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.335221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.335312 (+ 91us) server_negotiation.cc:407] Connection header received
0924 15:30:12.335396 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.335397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.335412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.335471 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:12.335634 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.336210 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337276 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.337709 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337805 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.337814 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.337978 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.338022 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:30:12.338486 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "5a4f579c339c428196963a775d031791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fa8e25bc554cf18356f839ad85cd54" is_pre_election: true
I20260924 15:30:12.338591 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.335062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.5.98.193:49919)
0924 15:30:12.335191 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.335206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.335292 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.335475 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:12.335476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:12.335478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:12.335539 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.335543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.336304 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.336306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.337172 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:12.337175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.337806 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.337807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:12.337880 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.337884 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:12.337969 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:30:12.338025 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260924 15:30:12.338665 20246 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4f579c339c428196963a775d031791 in term 0.
I20260924 15:30:12.338930 19903 leader_election.cc:302] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4f579c339c428196963a775d031791, b4fa8e25bc554cf18356f839ad85cd54; no voters:
I20260924 15:30:12.339071 20021 raft_consensus.cc:2781] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:12.339138 20021 raft_consensus.cc:482] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:12.339157 20021 raft_consensus.cc:3037] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:12.339087 20111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "5a4f579c339c428196963a775d031791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f30010af414ab3858c15e26506c960" is_pre_election: true
I20260924 15:30:12.339267 20111 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4f579c339c428196963a775d031791 in term 0.
I20260924 15:30:12.340097 20021 raft_consensus.cc:504] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.340250 20021 leader_election.cc:288] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [CANDIDATE]: Term 1 election: Requested vote from peers 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233), b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347)
I20260924 15:30:12.340397 20111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "5a4f579c339c428196963a775d031791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f30010af414ab3858c15e26506c960"
I20260924 15:30:12.340451 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "5a4f579c339c428196963a775d031791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fa8e25bc554cf18356f839ad85cd54"
I20260924 15:30:12.340515 20246 raft_consensus.cc:3037] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:12.340505 20111 raft_consensus.cc:3037] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:12.341450 20111 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4f579c339c428196963a775d031791 in term 1.
I20260924 15:30:12.341450 20246 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4f579c339c428196963a775d031791 in term 1.
I20260924 15:30:12.341629 19903 leader_election.cc:302] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4f579c339c428196963a775d031791, b4fa8e25bc554cf18356f839ad85cd54; no voters:
I20260924 15:30:12.341727 20021 raft_consensus.cc:2781] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:12.341838 20021 raft_consensus.cc:686] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4f579c339c428196963a775d031791, State: Running, Role: LEADER
I20260924 15:30:12.341892 20021 consensus_queue.cc:237] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.342710 19825 catalog_manager.cc:5495] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4f579c339c428196963a775d031791 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "5a4f579c339c428196963a775d031791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } health_report { overall_health: HEALTHY } } }
I20260924 15:30:12.390072 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.387478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.5.98.193:42865)
0924 15:30:12.387676 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:12.387678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:12.387685 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:30:12.387737 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:12.387737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:12.387758 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:12.387855 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:12.387996 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.388506 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.389258 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:12.389462 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:12.389538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:12.389710 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:12.389820 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:12.389904 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:12.389942 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:30:12.389964 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260924 15:30:12.391635 20111 raft_consensus.cc:1261] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Refusing update from remote peer 5a4f579c339c428196963a775d031791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:12.391752 20246 raft_consensus.cc:1261] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Refusing update from remote peer 5a4f579c339c428196963a775d031791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:12.392030 20021 consensus_queue.cc:1035] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:12.392217 20310 consensus_queue.cc:1035] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:12.394521 20314 mvcc.cc:205] Tried to move back new op lower bound from 7332920575553445888 to 7332920575353135104. Current Snapshot: MvccSnapshot[applied={T|T < 7332920575553445888}]
I20260924 15:30:12.394712 20316 mvcc.cc:205] Tried to move back new op lower bound from 7332920575553445888 to 7332920575353135104. Current Snapshot: MvccSnapshot[applied={T|T < 7332920575553445888}]
I20260924 15:30:12.395226 20315 mvcc.cc:205] Tried to move back new op lower bound from 7332920575553445888 to 7332920575353135104. Current Snapshot: MvccSnapshot[applied={T|T < 7332920575553445888 or (T in {7332920575553445888})}]
I20260924 15:30:12.396730 19824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:12.396811 19824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:12.397554 19824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Sending DeleteTablet for 3 replicas of tablet 0d10ba6e49064effb8bfb01a1a3fd8f0
I20260924 15:30:12.397850 19949 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:12 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:41278
I20260924 15:30:12.397893 20090 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:12 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:42520
I20260924 15:30:12.397965 20225 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:12 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:43880
I20260924 15:30:12.398090 20324 tablet_replica.cc:330] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: stopping tablet replica
I20260924 15:30:12.398186 20324 raft_consensus.cc:2229] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:30:12.398196 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19863
I20260924 15:30:12.398334 20324 raft_consensus.cc:2258] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:12.399437 20325 tablet_replica.cc:330] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: stopping tablet replica
I20260924 15:30:12.399544 20325 raft_consensus.cc:2229] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:12.399649 20325 raft_consensus.cc:2258] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:12.399756 20326 tablet_replica.cc:330] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: stopping tablet replica
I20260924 15:30:12.399860 20326 raft_consensus.cc:2229] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:12.399968 20326 raft_consensus.cc:2258] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:12.400429 20326 ts_tablet_manager.cc:1905] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:12.401957 20326 ts_tablet_manager.cc:1918] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:12.402050 20326 log.cc:1196] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/0d10ba6e49064effb8bfb01a1a3fd8f0
I20260924 15:30:12.402339 20326 ts_tablet_manager.cc:1939] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Deleting consensus metadata
I20260924 15:30:12.402747 19812 catalog_manager.cc:4841] TS b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347): tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 (table test-table [id=5f180bcfd39f48818f5b8bb1674459ea]) successfully deleted
W20260924 15:30:12.404356 19810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:42865 (error 108)
I20260924 15:30:12.404515 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20020
I20260924 15:30:12.404567 20325 ts_tablet_manager.cc:1905] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:12.405543 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.405407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.0.0.1:41300)
0924 15:30:12.405464 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.405507 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:42865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:30:12.405714 19810 catalog_manager.cc:4584] TS 5a4f579c339c428196963a775d031791 (127.5.98.193:42865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0: Network error: Client connection negotiation failed: client connection to 127.5.98.193:42865: connect: Connection refused (error 111)
I20260924 15:30:12.409296 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20160
I20260924 15:30:12.409368 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.409258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.0.0.1:42534)
0924 15:30:12.409311 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.409336 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.194:46233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:30:12.414122 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 19791
I20260924 15:30:12.419458 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--webserver_interface=127.5.98.254
--webserver_port=40701
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:12.501345 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:12.501547 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:12.501570 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:12.502638 20328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:12.502669 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:12.502683 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:12.502696 20328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:12.502707 20328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:12.504217 20328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=40701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:12.504421 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:12.504607 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:12.509479 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.509536 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.509562 20328 server_base.cc:1034] running on GCE node
W20260924 15:30:12.509479 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.510048 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:12.510308 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:12.511449 20328 hybrid_clock.cc:648] HybridClock initialized: now 1790263812511438 us; error 33 us; skew 500 ppm
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:12.513988 20328 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:12.514423 20328 webserver.cc:460] Webserver started at http://127.5.98.254:40701/ using document root <none> and password file <none>
I20260924 15:30:12.514672 20328 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:12.514720 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:12.516101 20328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 15:30:12.516790 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.516997 20328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:12.517104 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "06913431cd9d4f04bb26a0a654ea16b1"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "8b59e8093e8fd289f1b1f30a48488c45"
server_key_iv: "e867212ef542122800087151fa5f647d"
server_key_version: "encryptionkey@0"
I20260924 15:30:12.517453 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:12.530661 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:12.532713 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:12.532877 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:12.536938 20328 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34033
I20260924 15:30:12.537015 20397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34033 every 8 connection(s)
I20260924 15:30:12.537395 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:12.539044 20398 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:30:12.539788 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap starting.
I20260924 15:30:12.544404 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20328
I20260924 15:30:12.544749 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:42865
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=42915
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:12.548664 20398 log.cc:824] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:12.550948 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:12.551245 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1: Bootstrap complete.
I20260924 15:30:12.553586 20398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:12.553807 20398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.553977 20398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:12.554054 20398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:12.554083 20398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:12.554150 20398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:30:12.555156 20398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:12.555243 20398 leader_election.cc:302] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06913431cd9d4f04bb26a0a654ea16b1; no voters:
I20260924 15:30:12.555465 20398 leader_election.cc:288] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:30:12.555501 20402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:30:12.555606 20402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [term 3 LEADER]: Becoming Leader. State: Replica: 06913431cd9d4f04bb26a0a654ea16b1, State: Running, Role: LEADER
I20260924 15:30:12.555723 20402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } }
I20260924 15:30:12.555752 20398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:12.556209 20403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:12.556210 20402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06913431cd9d4f04bb26a0a654ea16b1. Latest consensus state: current_term: 3 leader_uuid: "06913431cd9d4f04bb26a0a654ea16b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06913431cd9d4f04bb26a0a654ea16b1" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34033 } } }
I20260924 15:30:12.556365 20402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:12.556316 20403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06913431cd9d4f04bb26a0a654ea16b1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:12.556556 20409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:12.557425 20409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:12.557523 20409 catalog_manager.cc:1259] Loaded cluster ID: 196d8d2942ac431f9fb57c201455709e
I20260924 15:30:12.557548 20409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 15:30:12.557754 20409 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f77b2f769ad google::LogMessage::Fail() at ??:0
@ 0x7f77b2f7988a google::LogMessage::SendToLog() at ??:0
@ 0x7f77b2f764de google::LogMessage::Flush() at ??:0
@ 0x7f77b2f77a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f77bae970f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f77baea08b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f77b4e440d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f77b4e39954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f77b5c9d6db start_thread at ??:0
@ 0x7f77b1a7461f clone at ??:0
W20260924 15:30:12.652431 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:12.652626 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:12.652649 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:12.653776 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:12.653816 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:12.653842 20400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:12.655303 20400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:42865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=42915
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:12.655548 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:12.655815 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:30:12.661967 20400 server_base.cc:1034] running on GCE node
W20260924 15:30:12.661993 20435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.662066 20434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.662072 20437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.662288 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:12.662569 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:12.663719 20400 hybrid_clock.cc:648] HybridClock initialized: now 1790263812663705 us; error 28 us; skew 500 ppm
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:12.666237 20400 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:12.666728 20400 webserver.cc:460] Webserver started at http://127.5.98.193:42915/ using document root <none> and password file <none>
I20260924 15:30:12.666975 20400 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:12.667047 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:12.668730 20400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:12.669481 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.669653 20400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:12.669757 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "5a4f579c339c428196963a775d031791"
format_stamp: "Formatted at 2026-09-24 15:30:10 on dist-test-slave-4g6f"
server_key: "31d42271449cd249147d856a20140639"
server_key_iv: "c57b0c13c1fd2ace99edfb2ea1cc1695"
server_key_version: "encryptionkey@0"
I20260924 15:30:12.670126 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:12.685194 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:12.687454 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:12.687628 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:12.687937 20400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:12.688387 20451 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:30:12.690374 20400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:30:12.690438 20400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:30:12.690480 20400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:30:12.691145 20400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:30:12.691202 20400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:12.691277 20451 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Bootstrap starting.
I20260924 15:30:12.698563 20400 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:42865
I20260924 15:30:12.698707 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:42865 every 8 connection(s)
I20260924 15:30:12.698984 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:30:12.700349 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20400
I20260924 15:30:12.700610 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:46233
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=44741
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:12.702679 20451 log.cc:824] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:12.705438 20451 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:12.705734 20451 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Bootstrap complete.
I20260924 15:30:12.706977 20451 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260924 15:30:12.708411 20451 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.708742 20451 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4f579c339c428196963a775d031791, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.708917 20451 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.709208 20451 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260924 15:30:12.764212 20563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:30:12.699758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:42923)
0924 15:30:12.699920 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.699959 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:12.700597 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:12.763962 (+ 63365us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260924 15:30:12.764453 20561 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:34033 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:34033: Client connection negotiation failed: client connection to 127.5.98.254:34033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:30:12.764812 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.764681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:34747)
0924 15:30:12.764745 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.764792 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:30:12.765201 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.765110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:49499)
0924 15:30:12.765160 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.765184 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:30:12.765316 20561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:30:12.785823 20564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:12.786051 20564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:12.786090 20564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:12.787243 20564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:12.787308 20564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:12.787353 20564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:12.788862 20564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:46233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=44741
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:12.789134 20564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:12.789361 20564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:12.794612 20571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.794641 20572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.794659 20574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.794677 20564 server_base.cc:1034] running on GCE node
I20260924 15:30:12.795030 20564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:12.795281 20564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:12.796434 20564 hybrid_clock.cc:648] HybridClock initialized: now 1790263812796420 us; error 28 us; skew 500 ppm
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:12.798817 20564 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:12.799281 20564 webserver.cc:460] Webserver started at http://127.5.98.194:44741/ using document root <none> and password file <none>
I20260924 15:30:12.799463 20564 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:12.799504 20564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:12.800846 20564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:12.801534 20581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.801726 20564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:12.801796 20564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "64f30010af414ab3858c15e26506c960"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "5e84504c85eee465ff3111da3d2ad118"
server_key_iv: "3997f370881d99450ccd2aa8aca9bf80"
server_key_version: "encryptionkey@0"
I20260924 15:30:12.802125 20564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:12.829506 20564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:12.831895 20564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:12.832011 20564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:12.832226 20564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:12.832752 20588 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:30:12.834338 20564 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:30:12.834403 20564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:30:12.834450 20564 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:30:12.835155 20564 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:30:12.835213 20564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:12.835283 20588 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Bootstrap starting.
I20260924 15:30:12.837059 20588 log.cc:824] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:12.839320 20588 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:12.839633 20588 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Bootstrap complete.
I20260924 15:30:12.839836 20588 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 15:30:12.841138 20588 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.841391 20588 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f30010af414ab3858c15e26506c960, State: Initialized, Role: FOLLOWER
I20260924 15:30:12.841573 20588 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:12.841883 20588 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:30:12.842602 20564 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:46233
I20260924 15:30:12.842674 20700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:46233 every 8 connection(s)
I20260924 15:30:12.842978 20564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:30:12.843977 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.843616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:56141)
0924 15:30:12.843799 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.843864 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
W20260924 15:30:12.844162 20701 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:34033 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:34033: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
I20260924 15:30:12.844444 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.844354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:48685)
0924 15:30:12.844401 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.844427 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:30:12.844699 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.844650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:54291)
0924 15:30:12.844670 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.844687 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 15:30:12.844805 20701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:30:12.846093 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20564
I20260924 15:30:12.846316 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:46347
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:34033
--webserver_port=37165
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:12.928212 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:12.928411 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:12.928434 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:12.929617 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:12.929661 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:12.929687 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:12.931207 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:46347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=37165
--tserver_master_addrs=127.5.98.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:12.931396 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:12.931593 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:12.936717 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.936712 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:12.936707 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:12.936966 20704 server_base.cc:1034] running on GCE node
I20260924 15:30:12.937181 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:12.937426 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:12.938567 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790263812938555 us; error 26 us; skew 500 ppm
Sep 24 15:30:12 dist-test-slave-4g6f krb5kdc[19240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:12.941121 20704 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:12.941520 20704 webserver.cc:460] Webserver started at http://127.5.98.195:37165/ using document root <none> and password file <none>
I20260924 15:30:12.941720 20704 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:12.941761 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:12.943022 20704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:12.943573 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.943756 20704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.943850 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "b4fa8e25bc554cf18356f839ad85cd54"
format_stamp: "Formatted at 2026-09-24 15:30:11 on dist-test-slave-4g6f"
server_key: "0025ca26dfb24ad1196c1bfa591bdac6"
server_key_iv: "fe122a0f9c4764e44ae683476a1e9a62"
server_key_version: "encryptionkey@0"
I20260924 15:30:12.944216 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:12.959566 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:12.962021 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:12.962206 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:12.962455 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:12.962777 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:12.962831 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.962872 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:12.962899 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:12.968951 20704 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46347
I20260924 15:30:12.968999 20834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46347 every 8 connection(s)
I20260924 15:30:12.969359 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:12.970429 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.970038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:53487)
0924 15:30:12.970199 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.970276 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
W20260924 15:30:12.970860 20835 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:34033 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:34033: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
I20260924 15:30:12.971194 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.971068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:57957)
0924 15:30:12.971138 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.971171 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 15:30:12.971453 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20704
I20260924 15:30:12.971691 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:12.971572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:33723)
0924 15:30:12.971611 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:12.971636 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 15:30:12.971819 20835 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:30:13.765915 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:13.765657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:59057)
0924 15:30:13.765779 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:13.765815 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260924 15:30:13.845412 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:13.845175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:46761)
0924 15:30:13.845312 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:13.845350 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260924 15:30:13.972378 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:13.972166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:37011)
0924 15:30:13.972278 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:13.972318 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260924 15:30:14.344672 20842 raft_consensus.cc:482] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:14.344790 20842 raft_consensus.cc:504] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:14.345181 20842 leader_election.cc:288] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347), 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
Sep 24 15:30:14 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.195@KRBTEST.COM
Sep 24 15:30:14 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:30:14.356786 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.345484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:36019 (local address 127.5.98.193:42865)
0924 15:30:14.345611 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:14.345615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:14.345628 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:30:14.345805 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:14.345807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:14.345831 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:14.345926 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:14.346669 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.348095 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.349417 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.350010 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.350132 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:14.353345 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:14.353348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:14.353348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:14.353367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:14.355081 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355505 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:14.355508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:14.355509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:14.355599 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355907 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:14.355908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:14.355909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:14.356140 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:14.356235 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:14.356521 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 15:30:14.356676 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":48,"threads_started":1}
I20260924 15:30:14.356802 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.345307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46347 (local address 127.5.98.194:51025)
0924 15:30:14.345424 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.345450 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:14.345633 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:14.346415 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:14.346416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:14.347066 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:14.347120 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:14.347122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.348544 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.348545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:14.349313 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:14.349315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.350136 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.350137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:14.350330 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:14.350971 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:14.350976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:14.352927 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:14.355071 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:14.355080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:14.355411 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:14.355691 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:14.355692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:14.355820 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:14.356217 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:14.356224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:14.356346 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 15:30:14.356457 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:30:14.356683 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":7}
I20260924 15:30:14.356812 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.345415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:51025 (local address 127.5.98.195:46347)
0924 15:30:14.345580 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:14.345593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:14.345630 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:30:14.346253 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:14.346255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:14.346292 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:14.346395 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:14.347223 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.348457 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.349389 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.350010 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.350128 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:14.353056 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:14.353061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:14.353061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:14.353074 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:14.354971 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355501 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:14.355504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:14.355505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:14.355599 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355895 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:14.355897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:14.355897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:14.356140 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:14.356235 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:14.356520 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 15:30:14.356670 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260924 15:30:14.357431 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.345345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:42865 (local address 127.5.98.194:36019)
0924 15:30:14.345475 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.345484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:14.345633 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:14.345976 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:14.345980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:14.346249 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:14.346545 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:14.346555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.348188 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.348189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:14.349284 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:14.349286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:14.350138 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:14.350139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:14.350330 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:14.350964 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:14.350976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:14.353243 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:14.355171 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:14.355172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:14.355410 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:14.355689 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:14.355690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:14.355690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:14.355818 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:14.356218 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:14.356225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:14.356348 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 15:30:14.356455 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:30:14.356768 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":179,"thread_start_us":68,"threads_started":1}
W20260924 15:30:14.357704 20585 leader_election.cc:341] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347): Not found: Tablet not found: 0d10ba6e49064effb8bfb01a1a3fd8f0
I20260924 15:30:14.357959 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "64f30010af414ab3858c15e26506c960" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5a4f579c339c428196963a775d031791" is_pre_election: true
I20260924 15:30:14.358106 20514 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f30010af414ab3858c15e26506c960 in term 1.
I20260924 15:30:14.358378 20583 leader_election.cc:302] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4f579c339c428196963a775d031791, 64f30010af414ab3858c15e26506c960; no voters: b4fa8e25bc554cf18356f839ad85cd54
I20260924 15:30:14.358497 20842 raft_consensus.cc:2781] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 15:30:14.358557 20842 raft_consensus.cc:482] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:14.358580 20842 raft_consensus.cc:3037] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:30:14.359747 20842 raft_consensus.cc:504] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:14.359905 20842 leader_election.cc:288] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 election: Requested vote from peers b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347), 5a4f579c339c428196963a775d031791 (127.5.98.193:42865)
I20260924 15:30:14.360093 20514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d10ba6e49064effb8bfb01a1a3fd8f0" candidate_uuid: "64f30010af414ab3858c15e26506c960" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5a4f579c339c428196963a775d031791"
I20260924 15:30:14.360172 20514 raft_consensus.cc:3037] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 1 FOLLOWER]: Advancing to term 2
W20260924 15:30:14.360209 20585 leader_election.cc:341] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347): Not found: Tablet not found: 0d10ba6e49064effb8bfb01a1a3fd8f0
I20260924 15:30:14.361245 20514 raft_consensus.cc:2454] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f30010af414ab3858c15e26506c960 in term 2.
I20260924 15:30:14.361471 20583 leader_election.cc:302] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a4f579c339c428196963a775d031791, 64f30010af414ab3858c15e26506c960; no voters: b4fa8e25bc554cf18356f839ad85cd54
I20260924 15:30:14.361563 20842 raft_consensus.cc:2781] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:30:14.361686 20842 raft_consensus.cc:686] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [term 2 LEADER]: Becoming Leader. State: Replica: 64f30010af414ab3858c15e26506c960, State: Running, Role: LEADER
I20260924 15:30:14.361797 20842 consensus_queue.cc:237] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:14.362011 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.361922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:57751)
0924 15:30:14.361957 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.361985 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 15:30:14.766786 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:34055)
0924 15:30:14.766677 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.766721 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
W20260924 15:30:14.881376 20585 consensus_peers.cc:487] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 -> Peer b4fa8e25bc554cf18356f839ad85cd54 (127.5.98.195:46347): Couldn't send request to peer b4fa8e25bc554cf18356f839ad85cd54. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d10ba6e49064effb8bfb01a1a3fd8f0. This is attempt 1: this message will repeat every 5th retry.
I20260924 15:30:14.959438 20514 raft_consensus.cc:1261] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 5a4f579c339c428196963a775d031791 [term 2 FOLLOWER]: Refusing update from remote peer 64f30010af414ab3858c15e26506c960: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 15:30:14.959915 20850 consensus_queue.cc:1035] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 15:30:14.960966 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.960824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:48467)
0924 15:30:14.960889 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.960932 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:30:14.961971 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.961797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:40821)
0924 15:30:14.961876 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.961932 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64,"thread_start_us":47,"threads_started":1}
I20260924 15:30:14.961966 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.961865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:44615)
0924 15:30:14.961917 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.961942 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:30:14.973096 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:14.972849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:39037)
0924 15:30:14.973018 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:14.973058 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260924 15:30:15.401255 20858 ts_tablet_manager.cc:927] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Initiating tablet copy from peer 64f30010af414ab3858c15e26506c960 (127.5.98.194:46233)
I20260924 15:30:15.401633 20858 tablet_copy_client.cc:316] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: tablet copy: Beginning tablet copy session from remote peer at address 127.5.98.194:46233
Sep 24 15:30:15 dist-test-slave-4g6f krb5kdc[19240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263812, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.194@KRBTEST.COM
I20260924 15:30:15.409848 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.402051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:44455 (local address 127.5.98.194:46233)
0924 15:30:15.402206 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:15.402210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:15.402218 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:15.402242 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:15.402243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:15.402255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:15.402314 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:15.402831 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:15.403523 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:15.404598 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:15.404972 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:15.405033 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:15.407036 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:15.407044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:15.407044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:15.407051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:15.408680 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:15.409083 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:15.409086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:15.409086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:15.409139 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:15.409319 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:15.409319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:15.409320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:15.409452 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:15.409512 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:15.409708 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:30:15.409773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 15:30:15.409852 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.401945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:46233 (local address 127.5.98.195:44455)
0924 15:30:15.402005 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.402039 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:15.402099 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:15.402346 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:15.402353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:15.402688 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:15.402752 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:15.402754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:15.403616 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:15.403617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:15.404472 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:15.404474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:15.405047 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:15.405048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:15.405107 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:15.405627 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:15.405631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:15.406958 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:15.408771 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:15.408777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:15.408779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:15.409001 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:15.409202 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:15.409202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:15.409203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:15.409262 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:15.409512 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:15.409518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:15.409565 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:30:15.409652 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 15:30:15.409767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 15:30:15.410310 20673 tablet_copy_service.cc:140] P 64f30010af414ab3858c15e26506c960: Received BeginTabletCopySession request for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 from peer b4fa8e25bc554cf18356f839ad85cd54 ({username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:44455)
I20260924 15:30:15.410372 20673 tablet_copy_service.cc:161] P 64f30010af414ab3858c15e26506c960: Beginning new tablet copy session on tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 from peer b4fa8e25bc554cf18356f839ad85cd54 at {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:44455: session id = b4fa8e25bc554cf18356f839ad85cd54-0d10ba6e49064effb8bfb01a1a3fd8f0
I20260924 15:30:15.410851 20673 tablet_copy_source_session.cc:205] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P 64f30010af414ab3858c15e26506c960: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:30:15.411978 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d10ba6e49064effb8bfb01a1a3fd8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:15.414577 20858 tablet_copy_client.cc:791] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: tablet copy: Starting download of 0 data blocks...
I20260924 15:30:15.414738 20858 tablet_copy_client.cc:655] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: tablet copy: Starting download of 1 WAL segments...
I20260924 15:30:15.415639 20858 tablet_copy_client.cc:531] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:30:15.416493 20858 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Bootstrap starting.
I20260924 15:30:15.417929 20858 log.cc:824] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:15.420624 20858 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:15.420866 20858 tablet_bootstrap.cc:492] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Bootstrap complete.
I20260924 15:30:15.421015 20858 ts_tablet_manager.cc:1397] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 15:30:15.421947 20858 raft_consensus.cc:348] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:15.422124 20858 raft_consensus.cc:729] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fa8e25bc554cf18356f839ad85cd54, State: Initialized, Role: FOLLOWER
I20260924 15:30:15.422266 20858 consensus_queue.cc:260] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f30010af414ab3858c15e26506c960" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 46233 } } peers { permanent_uuid: "b4fa8e25bc554cf18356f839ad85cd54" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46347 } } peers { permanent_uuid: "5a4f579c339c428196963a775d031791" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 42865 } }
I20260924 15:30:15.422523 20858 ts_tablet_manager.cc:1428] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 15:30:15.422832 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.422714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:49127)
0924 15:30:15.422772 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.422802 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:30:15.422838 20673 tablet_copy_service.cc:342] P 64f30010af414ab3858c15e26506c960: Request end of tablet copy session b4fa8e25bc554cf18356f839ad85cd54-0d10ba6e49064effb8bfb01a1a3fd8f0 received from {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:44455
I20260924 15:30:15.422896 20673 tablet_copy_service.cc:433] P 64f30010af414ab3858c15e26506c960: ending tablet copy session b4fa8e25bc554cf18356f839ad85cd54-0d10ba6e49064effb8bfb01a1a3fd8f0 on tablet 0d10ba6e49064effb8bfb01a1a3fd8f0 with peer b4fa8e25bc554cf18356f839ad85cd54
I20260924 15:30:15.423185 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.423090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:51465)
0924 15:30:15.423137 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.423161 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:30:15.962584 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.962369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:39885)
0924 15:30:15.962508 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.962539 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":49,"threads_started":1}
I20260924 15:30:15.962584 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.962380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:54191)
0924 15:30:15.962506 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.962543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260924 15:30:15.999605 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:15.999461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:40135)
0924 15:30:15.999528 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:15.999567 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 15:30:15.999691 20788 raft_consensus.cc:1203] T 0d10ba6e49064effb8bfb01a1a3fd8f0 P b4fa8e25bc554cf18356f839ad85cd54 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 15:30:16.000311 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:16.000156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:47473)
0924 15:30:16.000249 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:16.000281 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
I20260924 15:30:16.963306 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:16.963089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:36457)
0924 15:30:16.963221 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:16.963258 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260924 15:30:17.000283 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:17.000052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:36723)
0924 15:30:17.000196 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:17.000237 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260924 15:30:17.000821 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:17.000635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:40249)
0924 15:30:17.000744 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:17.000775 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260924 15:30:17.964135 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:17.963930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:44721)
0924 15:30:17.964053 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:17.964089 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260924 15:30:18.001004 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:18.000787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:39285)
0924 15:30:18.000919 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:18.000957 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 15:30:18.001519 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:18.001317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:51701)
0924 15:30:18.001444 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:18.001481 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260924 15:30:18.964768 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:18.964569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:45235)
0924 15:30:18.964688 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:18.964724 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 15:30:19.001720 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:19.001500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:46295)
0924 15:30:19.001648 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:19.001677 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260924 15:30:19.002134 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:19.001920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:47913)
0924 15:30:19.002052 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:19.002088 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":45,"threads_started":1}
I20260924 15:30:19.965517 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:19.965263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:36099)
0924 15:30:19.965430 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:19.965468 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260924 15:30:20.002633 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:20.002387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:54587)
0924 15:30:20.002526 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:20.002570 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260924 15:30:20.002848 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:20.002589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:49137)
0924 15:30:20.002749 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:20.002791 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260924 15:30:20.966346 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:20.966135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:34019)
0924 15:30:20.966256 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:20.966299 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260924 15:30:21.003337 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:21.003100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:60711)
0924 15:30:21.003250 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:21.003288 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260924 15:30:21.003592 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:21.003347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:41845)
0924 15:30:21.003517 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:21.003552 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260924 15:30:21.967065 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:21.966855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:46451)
0924 15:30:21.966974 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:21.967010 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260924 15:30:22.003999 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:22.003779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:40321)
0924 15:30:22.003915 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:22.003953 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 15:30:22.004302 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:22.004077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:47835)
0924 15:30:22.004224 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:22.004266 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260924 15:30:22.967839 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:22.967590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:42963)
0924 15:30:22.967755 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:22.967792 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260924 15:30:23.004645 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:23.004420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:48895)
0924 15:30:23.004550 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:23.004583 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260924 15:30:23.004961 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:23.004702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:58407)
0924 15:30:23.004858 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:23.004900 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260924 15:30:23.968483 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:23.968268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:46383)
0924 15:30:23.968402 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:23.968437 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260924 15:30:24.005407 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:24.005147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:60581)
0924 15:30:24.005294 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:24.005347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260924 15:30:24.005681 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:24.005473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:60371)
0924 15:30:24.005604 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:24.005641 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 15:30:24.969213 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:24.969004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:37645)
0924 15:30:24.969131 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:24.969167 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260924 15:30:25.006134 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:25.005823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:43561)
0924 15:30:25.005991 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:25.006036 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260924 15:30:25.006311 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:25.006085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:36659)
0924 15:30:25.006209 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:25.006254 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260924 15:30:25.969872 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:25.969634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:44219)
0924 15:30:25.969791 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:25.969827 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260924 15:30:26.006992 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:26.006720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:44373)
0924 15:30:26.006876 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:26.006931 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260924 15:30:26.007107 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:26.006834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:44613)
0924 15:30:26.007013 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:26.007055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260924 15:30:26.970582 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:26.970370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:40861)
0924 15:30:26.970500 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:26.970540 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260924 15:30:27.007709 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:27.007444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.194:40515)
0924 15:30:27.007576 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:27.007616 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 15:30:27.007879 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:27.007584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.195:56491)
0924 15:30:27.007757 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:27.007814 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260924 15:30:27.971266 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:27.971055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34033 (local address 127.5.98.193:59915)
0924 15:30:27.971192 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:27.971228 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260924 15:30:27.973827 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20400
I20260924 15:30:27.979436 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20564
I20260924 15:30:27.984091 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20704
2026-09-24T15:30:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18439 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:30:28 dist-test-slave-4g6f krb5kdc[20906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:30:28 dist-test-slave-4g6f krb5kdc[20906](info): set up 2 sockets
Sep 24 15:30:28 dist-test-slave-4g6f krb5kdc[20906](info): commencing operation
krb5kdc: starting...
W20260924 15:30:29.028671 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:30:29.147842 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:29.229197 20919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:29.229405 20919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:29.229429 20919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:29.230496 20919 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:30:29.230530 20919 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:30:29.230543 20919 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:30:29.230556 20919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:29.230567 20919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:29.230580 20919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:29.230592 20919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:29.230604 20919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:29.232044 20919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:29.232237 20919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:29.232409 20919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:29.237454 20924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.237471 20925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.237471 20927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:29.237545 20919 server_base.cc:1034] running on GCE node
I20260924 15:30:29.237998 20919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:29.238297 20919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:29.239439 20919 hybrid_clock.cc:648] HybridClock initialized: now 1790263829239417 us; error 42 us; skew 500 ppm
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:29.242027 20919 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:29.242486 20919 webserver.cc:460] Webserver started at http://127.5.98.254:45495/ using document root <none> and password file <none>
I20260924 15:30:29.242727 20919 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:29.242798 20919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:29.242950 20919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:29.244089 20919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "978a301706ca4e05be8ce733502a3ade"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "33272b4b403e5f386d30b2e2454c895d"
server_key_iv: "54fffaffb2762f14a5fe59b73ad4b56f"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.244457 20919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "978a301706ca4e05be8ce733502a3ade"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "33272b4b403e5f386d30b2e2454c895d"
server_key_iv: "54fffaffb2762f14a5fe59b73ad4b56f"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.245805 20919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:29.246541 20934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.246721 20919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:29.246829 20919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "978a301706ca4e05be8ce733502a3ade"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "33272b4b403e5f386d30b2e2454c895d"
server_key_iv: "54fffaffb2762f14a5fe59b73ad4b56f"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.246906 20919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:29.271833 20919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:29.274267 20919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:29.274438 20919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:29.278582 20919 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36519
I20260924 15:30:29.278644 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36519 every 8 connection(s)
I20260924 15:30:29.279004 20919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:29.279927 20989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:29.282161 20989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap starting.
I20260924 15:30:29.282699 20989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:29.282860 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 20919
I20260924 15:30:29.282980 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:30:29.283070 20989 log.cc:824] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:29.283214 5515 external_mini_cluster.cc:1428] Setting key 190d01616a147512471a98c86f66a377
I20260924 15:30:29.283986 20989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: No bootstrap required, opened a new log
I20260924 15:30:29.286026 20989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:29.286206 20989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:29.286245 20989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Initialized, Role: FOLLOWER
I20260924 15:30:29.286410 20989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:29.286494 20989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:29.286525 20989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:29.286569 20989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:29.287243 20989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:29.287360 20989 leader_election.cc:302] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978a301706ca4e05be8ce733502a3ade; no voters:
I20260924 15:30:29.287511 20989 leader_election.cc:288] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:30:29.287580 20994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:29.287747 20989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:29.287758 20994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 LEADER]: Becoming Leader. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Running, Role: LEADER
I20260924 15:30:29.288079 20994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:29.288465 20996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:30:29.288543 20996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:29.288733 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978a301706ca4e05be8ce733502a3ade. Latest consensus state: current_term: 1 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:30:29.288743 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:29.288858 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:29.289655 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:29.291364 21003 catalog_manager.cc:1347] Generated new cluster ID: e7b9a436d0624dea880c39a58d1cc197
I20260924 15:30:29.291419 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:29.295800 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.283928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.5.98.254:36519)
0924 15:30:29.284129 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.284140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.284182 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:30:29.285223 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.285232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.285295 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.285453 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:29.286025 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.287802 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.289031 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.289615 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.289747 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.292582 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:29.292586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:29.292586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:29.292597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:29.294264 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.294625 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.294628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.294628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.294727 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.294936 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.294938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.294938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.295177 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:29.295339 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.295448 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.295581 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260924 15:30:29.296939 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:30:29.297405 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:29.301177 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Generated new TSK 0
I20260924 15:30:29.301331 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:30:29.334429 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:29.436518 21015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:29.436748 21015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:29.436787 21015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:29.437934 21015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:29.438000 21015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:29.438046 21015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:29.439481 21015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:29.439779 21015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:29.440023 21015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:29.445240 21021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.445287 21020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.445222 21023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:29.445561 21015 server_base.cc:1034] running on GCE node
I20260924 15:30:29.445890 21015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:29.446189 21015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:29.447371 21015 hybrid_clock.cc:648] HybridClock initialized: now 1790263829447351 us; error 33 us; skew 500 ppm
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:29.450343 21015 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:29.450855 21015 webserver.cc:460] Webserver started at http://127.5.98.193:38927/ using document root <none> and password file <none>
I20260924 15:30:29.451097 21015 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:29.451165 21015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:29.451328 21015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:29.452502 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "1394ccabc0804aebbec5a1f667c92d77"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "8a9f2037640b3bc7f1827ba0a12cc702"
server_key_iv: "e2a31bef7326729fa0ed99614e0b2e53"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.452884 21015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "1394ccabc0804aebbec5a1f667c92d77"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "8a9f2037640b3bc7f1827ba0a12cc702"
server_key_iv: "e2a31bef7326729fa0ed99614e0b2e53"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.454254 21015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:30:29.454984 21030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.455170 21015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:29.455269 21015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "1394ccabc0804aebbec5a1f667c92d77"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "8a9f2037640b3bc7f1827ba0a12cc702"
server_key_iv: "e2a31bef7326729fa0ed99614e0b2e53"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.455348 21015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:29.481091 21015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:29.483094 21015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:29.483268 21015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:29.483520 21015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:29.483868 21015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:29.483923 21015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.483959 21015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:29.483989 21015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.489934 21015 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:36281
I20260924 15:30:29.489984 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:36281 every 8 connection(s)
I20260924 15:30:29.490341 21015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:29.499425 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.491083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:46873 (local address 127.5.98.254:36519)
0924 15:30:29.491134 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.491137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.491329 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:30:29.491893 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.491894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.491907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.491955 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:29.492952 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.493479 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.494490 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.494705 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.494784 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.497031 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:29.497033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:29.497033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:29.497040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:29.498235 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.498599 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.498600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.498600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.498650 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.498864 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.498864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.498864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.498977 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:29.499028 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.499276 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.499350 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:30:29.499557 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.491043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:46873)
0924 15:30:29.491191 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.491239 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.491779 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.492027 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.492031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.492635 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:29.492837 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.492842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.493577 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.493578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.494394 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.494395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.494791 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.494792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.494936 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.495506 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:29.495515 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:29.496940 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:29.498308 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.498315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.498316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.498531 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.498710 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.498711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.498711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.498808 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.499042 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:29.499047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:29.499162 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.499244 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.499364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260924 15:30:29.500172 21146 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:29.500298 21146 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:29.500463 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21015
I20260924 15:30:29.500491 21146 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:29.500559 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:30:29.500835 5515 external_mini_cluster.cc:1428] Setting key a0b50a1d4e2111eddba8518a8b06ed28
I20260924 15:30:29.501068 20951 ts_manager.cc:194] Registered new tserver with Master: 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281)
I20260924 15:30:29.501902 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:46873
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:30:29.527110 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:29.608734 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:29.608924 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:29.608947 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:29.610082 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:29.610122 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:29.610148 21151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:29.611569 21151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:29.611806 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:29.612067 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:29.616997 21156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.616981 21157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.616971 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:29.617259 21151 server_base.cc:1034] running on GCE node
I20260924 15:30:29.617456 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:29.617650 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:29.618871 21151 hybrid_clock.cc:648] HybridClock initialized: now 1790263829618859 us; error 27 us; skew 500 ppm
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:29.621402 21151 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:29.621820 21151 webserver.cc:460] Webserver started at http://127.5.98.194:40663/ using document root <none> and password file <none>
I20260924 15:30:29.622069 21151 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:29.622116 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:29.622262 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:29.623301 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "68e91b3e3cfa49069f3f63db85f95161"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "893488965e31a171fa742dde62fc6fa7"
server_key_iv: "ff8dbd1fce42bf6d4218d90acc2f9903"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.623713 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "68e91b3e3cfa49069f3f63db85f95161"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "893488965e31a171fa742dde62fc6fa7"
server_key_iv: "ff8dbd1fce42bf6d4218d90acc2f9903"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.625025 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:29.625793 21166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.625974 21151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:30:29.626062 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "68e91b3e3cfa49069f3f63db85f95161"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "893488965e31a171fa742dde62fc6fa7"
server_key_iv: "ff8dbd1fce42bf6d4218d90acc2f9903"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.626140 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:29.638569 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:29.640447 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:29.640605 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:29.640825 21151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:29.641129 21151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:29.641163 21151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.641185 21151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:29.641229 21151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.646958 21151 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40875
I20260924 15:30:29.647050 21281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40875 every 8 connection(s)
I20260924 15:30:29.647331 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:30:29.651906 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21151
I20260924 15:30:29.652019 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:30:29.652246 5515 external_mini_cluster.cc:1428] Setting key a31ea2bc741b8b5bd05e07f448d6458d
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:29.657850 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.648123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59593 (local address 127.5.98.254:36519)
0924 15:30:29.648182 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.648184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.648424 (+ 240us) server_negotiation.cc:407] Connection header received
0924 15:30:29.648969 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.648970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.648982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.649045 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:29.650062 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.650582 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.651646 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.651889 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.651965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.654723 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:29.654724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:29.654724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:29.654731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:29.656386 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.656809 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.656811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.656811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.656874 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.657216 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.657218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.657218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.657363 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:29.657439 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.657716 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.657789 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:30:29.658006 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.648090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:59593)
0924 15:30:29.648278 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.648314 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.648862 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.649119 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.649122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.649764 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:29.649963 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.649968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.650681 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.650683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.651542 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.651544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.651986 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.651987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.652152 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.652720 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:29.652730 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:29.654610 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:29.656484 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.656491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.656493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.656725 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.656993 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.656995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.656995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.657134 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.657437 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:29.657443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:29.657559 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.657652 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.657772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 15:30:29.658427 21282 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:29.658532 21282 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:29.658728 21282 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:29.659179 20951 ts_manager.cc:194] Registered new tserver with Master: 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
I20260924 15:30:29.659560 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:59593
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:30:29.679183 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:29.762712 21287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:29.762916 21287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:29.762948 21287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:29.764174 21287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:29.764214 21287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:29.764241 21287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:29.765803 21287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:29.766001 21287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:29.766175 21287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:29.771116 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.771126 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:29.771128 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:29.771495 21287 server_base.cc:1034] running on GCE node
I20260924 15:30:29.771768 21287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:29.772045 21287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:29.773388 21287 hybrid_clock.cc:648] HybridClock initialized: now 1790263829773359 us; error 31 us; skew 500 ppm
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:29.775910 21287 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:29.776329 21287 webserver.cc:460] Webserver started at http://127.5.98.195:38855/ using document root <none> and password file <none>
I20260924 15:30:29.776577 21287 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:29.776644 21287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:29.776791 21287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:30:29.777874 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "51983300742c49bb977204ee6b6068b7"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "5cad1220538475632907367959e8216b"
server_key_iv: "3bd78f43b7ae53fa3f9f4eb0a58e8ff0"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.778224 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "51983300742c49bb977204ee6b6068b7"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "5cad1220538475632907367959e8216b"
server_key_iv: "3bd78f43b7ae53fa3f9f4eb0a58e8ff0"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.779500 21287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:30:29.780292 21302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.780454 21287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:29.780551 21287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "51983300742c49bb977204ee6b6068b7"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "5cad1220538475632907367959e8216b"
server_key_iv: "3bd78f43b7ae53fa3f9f4eb0a58e8ff0"
server_key_version: "encryptionkey@0"
I20260924 15:30:29.780633 21287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:29.812270 21287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:29.814334 21287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:29.814491 21287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:29.814731 21287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:29.815034 21287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:29.815117 21287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.815165 21287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:29.815193 21287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:29.820925 21287 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42847
I20260924 15:30:29.820968 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42847 every 8 connection(s)
I20260924 15:30:29.821281 21287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:30:29.824191 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21287
I20260924 15:30:29.824298 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:30:29.824501 5515 external_mini_cluster.cc:1428] Setting key 7687380a79ae5f49032d1c5373c20b41
Sep 24 15:30:29 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:29.830462 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.821990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:50109 (local address 127.5.98.254:36519)
0924 15:30:29.822058 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.822060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.822269 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:30:29.822742 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.822743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.822753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.822806 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:29.823773 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.824258 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.825531 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.825727 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.825812 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.827922 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:29.827924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:29.827924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:29.827931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:29.829107 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.829536 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.829537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.829538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.829595 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.829903 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.829905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.829905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.830036 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:29.830093 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.830322 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.830383 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:30:29.830610 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.821970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:50109)
0924 15:30:29.822153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.822189 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.822661 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.822887 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.822891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.823441 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:29.823638 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.823642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.824377 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.824378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.825433 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.825436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.825842 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.825844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.825989 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.826413 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:29.826422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:29.827830 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:29.829201 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.829208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.829210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.829446 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.829664 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:29.829665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:29.829665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:29.829778 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:29.830095 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:29.830100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:29.830210 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.830293 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.830409 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260924 15:30:29.831043 21417 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:29.831140 21417 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:29.831328 21417 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:29.831760 20951 ts_manager.cc:194] Registered new tserver with Master: 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847)
I20260924 15:30:29.832132 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:50109
I20260924 15:30:29.836170 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:29.843123 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.837468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.5.98.254:36519)
0924 15:30:29.837520 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.837522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.837652 (+ 130us) server_negotiation.cc:407] Connection header received
0924 15:30:29.837694 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.837694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.837702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.837745 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:29.838241 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.838699 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.839428 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.839612 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.839721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.840772 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:29.840773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:29.840773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:29.840779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:29.842105 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.842485 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.842486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.842486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.842534 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:29.842718 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:29.842719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:29.842719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:29.842844 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:29.842916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.843010 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.843066 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:30:29.844762 20951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:30:29.846297 20951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:30:29.853001 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.848862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.5.98.195:42847)
0924 15:30:29.849045 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.849050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.849060 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:29.849102 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.849104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.849123 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.849198 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.849461 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850602 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.851864 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.852576 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.852713 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.852868 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.852908 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260924 15:30:29.853001 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.848634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.0.0.1:60006)
0924 15:30:29.849090 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.849099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.849134 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.849492 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.849493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.849495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.849539 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.849541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850486 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.850487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.851911 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.851914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.852785 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.852848 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.852851 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.852899 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.852928 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":43,"thread_start_us":75,"threads_started":1}
I20260924 15:30:29.853001 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.848634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42847 (local address 127.0.0.1:47414)
0924 15:30:29.848900 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.848913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.848981 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.849303 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.849307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.849309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.849370 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.849372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850710 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.850711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.851784 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.851786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.852665 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.852733 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.852764 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.852839 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.852916 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":46,"thread_start_us":80,"threads_started":1}
I20260924 15:30:29.853255 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.849061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.5.98.193:36281)
0924 15:30:29.849247 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.849252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.849263 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:30:29.849306 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.849308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.849325 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.849407 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.849622 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850384 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852013 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.852709 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852773 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.852846 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.853129 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.853178 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260924 15:30:29.854013 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.848607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40875 (local address 127.0.0.1:36200)
0924 15:30:29.848865 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.848906 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.848987 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.849528 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.849529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.849531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.849569 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.849570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850825 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.850826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.851933 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.851935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.853786 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.853787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.853848 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.853853 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.853906 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.853946 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":55,"threads_started":1}
I20260924 15:30:29.854305 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.848716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.5.98.194:40875)
0924 15:30:29.849303 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.849307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.849317 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:29.849366 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.849368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.849384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.849453 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.849635 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.850726 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.852880 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.853630 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.853695 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.853753 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.854187 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.854232 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":87,"threads_started":1}
I20260924 15:30:29.854192 21350 tablet_service.cc:1467] Processing CreateTablet for tablet 179921b3637c48ef92a5fe206cd114b6 (DEFAULT_TABLE table=test-table [id=abb94493152848d486a07b7b2d6c8f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:29.854192 21078 tablet_service.cc:1467] Processing CreateTablet for tablet 179921b3637c48ef92a5fe206cd114b6 (DEFAULT_TABLE table=test-table [id=abb94493152848d486a07b7b2d6c8f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:29.854547 21078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179921b3637c48ef92a5fe206cd114b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:29.854547 21350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179921b3637c48ef92a5fe206cd114b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:29.854874 21214 tablet_service.cc:1467] Processing CreateTablet for tablet 179921b3637c48ef92a5fe206cd114b6 (DEFAULT_TABLE table=test-table [id=abb94493152848d486a07b7b2d6c8f9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:29.855197 21214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179921b3637c48ef92a5fe206cd114b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:29.857759 21435 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Bootstrap starting.
I20260924 15:30:29.858201 21436 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap starting.
I20260924 15:30:29.858580 21435 tablet_bootstrap.cc:654] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:29.858836 21436 tablet_bootstrap.cc:654] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:29.859118 21435 log.cc:824] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:29.859438 21436 log.cc:824] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:29.859776 21437 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Bootstrap starting.
I20260924 15:30:29.859949 21435 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: No bootstrap required, opened a new log
I20260924 15:30:29.860020 21435 ts_tablet_manager.cc:1397] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:29.860432 21437 tablet_bootstrap.cc:654] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:29.861004 21437 log.cc:824] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:29.861557 21436 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: No bootstrap required, opened a new log
I20260924 15:30:29.861629 21436 ts_tablet_manager.cc:1397] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 15:30:29.861852 21437 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: No bootstrap required, opened a new log
I20260924 15:30:29.861933 21437 ts_tablet_manager.cc:1397] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:29.862213 21435 raft_consensus.cc:348] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.862413 21435 raft_consensus.cc:374] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:29.862452 21435 raft_consensus.cc:729] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e91b3e3cfa49069f3f63db85f95161, State: Initialized, Role: FOLLOWER
I20260924 15:30:29.862576 21435 consensus_queue.cc:260] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.862855 21435 ts_tablet_manager.cc:1428] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:30:29.862998 21282 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
I20260924 15:30:29.863628 21436 raft_consensus.cc:348] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.863835 21436 raft_consensus.cc:374] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:29.863888 21436 raft_consensus.cc:729] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1394ccabc0804aebbec5a1f667c92d77, State: Initialized, Role: FOLLOWER
I20260924 15:30:29.864006 21436 consensus_queue.cc:260] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.864151 21437 raft_consensus.cc:348] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.864289 21146 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
I20260924 15:30:29.864322 21437 raft_consensus.cc:374] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:29.864377 21437 raft_consensus.cc:729] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51983300742c49bb977204ee6b6068b7, State: Initialized, Role: FOLLOWER
I20260924 15:30:29.864485 21437 consensus_queue.cc:260] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.864524 21436 ts_tablet_manager.cc:1428] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:30:29.864782 21437 ts_tablet_manager.cc:1428] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:30:29.864784 21417 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
W20260924 15:30:29.897953 21283 tablet.cc:2369] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:30:29.923369 21442 raft_consensus.cc:482] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:29.923502 21442 raft_consensus.cc:504] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.923892 21442 leader_election.cc:288] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875), 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847)
I20260924 15:30:29.927249 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.924189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:59507 (local address 127.5.98.194:40875)
0924 15:30:29.924248 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.924250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.924259 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:30:29.924291 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.924292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.924305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.924370 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.924576 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.925087 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.926483 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.926846 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.926964 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.926975 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.927144 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.927184 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:30:29.927249 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.924060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40875 (local address 127.5.98.193:59507)
0924 15:30:29.924120 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.924131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.924178 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.924405 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.924407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.924410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.924498 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.924500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.925193 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.925194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.926395 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.926398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.926964 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.926965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.927052 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.927057 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.927126 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.927171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":4}
I20260924 15:30:29.927331 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.924195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:35005 (local address 127.5.98.195:42847)
0924 15:30:29.924263 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.924266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.924412 (+ 146us) server_negotiation.cc:407] Connection header received
0924 15:30:29.924439 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.924441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.924450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.924507 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.924704 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.925192 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.926521 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.926959 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.927054 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.927063 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.927218 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.927260 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:30:29.927582 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.924136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42847 (local address 127.5.98.193:35005)
0924 15:30:29.924309 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.924318 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.924369 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.924526 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.924527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.924528 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.924618 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.924619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.925257 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.925258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.926395 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.926398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.927061 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.927063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.927132 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.927136 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.927209 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.927250 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
I20260924 15:30:29.927747 21234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "1394ccabc0804aebbec5a1f667c92d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e91b3e3cfa49069f3f63db85f95161" is_pre_election: true
I20260924 15:30:29.927915 21234 raft_consensus.cc:2454] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1394ccabc0804aebbec5a1f667c92d77 in term 0.
I20260924 15:30:29.927955 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "1394ccabc0804aebbec5a1f667c92d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51983300742c49bb977204ee6b6068b7" is_pre_election: true
I20260924 15:30:29.928064 21443 raft_consensus.cc:482] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:29.928155 21031 leader_election.cc:302] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77, 68e91b3e3cfa49069f3f63db85f95161; no voters:
I20260924 15:30:29.928159 21443 raft_consensus.cc:504] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.928282 21442 raft_consensus.cc:2781] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:29.928326 21442 raft_consensus.cc:482] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:29.928332 21371 raft_consensus.cc:2454] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1394ccabc0804aebbec5a1f667c92d77 in term 0.
I20260924 15:30:29.928355 21442 raft_consensus.cc:3037] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:29.928642 21443 leader_election.cc:288] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875), 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281)
I20260924 15:30:29.929457 21442 raft_consensus.cc:504] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.929747 21234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "1394ccabc0804aebbec5a1f667c92d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e91b3e3cfa49069f3f63db85f95161"
I20260924 15:30:29.929849 21234 raft_consensus.cc:3037] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:29.930052 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "1394ccabc0804aebbec5a1f667c92d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51983300742c49bb977204ee6b6068b7"
I20260924 15:30:29.930186 21371 raft_consensus.cc:3037] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:29.930758 21234 raft_consensus.cc:2454] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1394ccabc0804aebbec5a1f667c92d77 in term 1.
I20260924 15:30:29.930933 21031 leader_election.cc:302] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77, 68e91b3e3cfa49069f3f63db85f95161; no voters:
I20260924 15:30:29.931012 21371 raft_consensus.cc:2454] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1394ccabc0804aebbec5a1f667c92d77 in term 1.
I20260924 15:30:29.929705 21442 leader_election.cc:288] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 1 election: Requested vote from peers 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875), 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847)
I20260924 15:30:29.931147 21442 raft_consensus.cc:2781] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:29.931193 21442 raft_consensus.cc:686] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 LEADER]: Becoming Leader. State: Replica: 1394ccabc0804aebbec5a1f667c92d77, State: Running, Role: LEADER
I20260924 15:30:29.931278 21442 consensus_queue.cc:237] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:29.932121 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.928631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40875 (local address 127.5.98.195:55507)
0924 15:30:29.928841 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.928851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.928884 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.929135 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.929137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.929139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.929195 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.929197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.930020 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.930021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.931150 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.931153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.931860 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.931861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.931933 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.931938 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.932003 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.932058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260924 15:30:29.932125 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.928839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:55507 (local address 127.5.98.194:40875)
0924 15:30:29.929017 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.929020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.929026 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:30:29.929045 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.929046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.929055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.929109 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.929274 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.929936 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.931238 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.931776 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.931848 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.931856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.932020 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.932056 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 15:30:29.932452 21234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "51983300742c49bb977204ee6b6068b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e91b3e3cfa49069f3f63db85f95161" is_pre_election: true
I20260924 15:30:29.932544 21234 raft_consensus.cc:2379] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51983300742c49bb977204ee6b6068b7 in current term 1: Already voted for candidate 1394ccabc0804aebbec5a1f667c92d77 in this term.
I20260924 15:30:29.932650 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.928665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:50891 (local address 127.5.98.193:36281)
0924 15:30:29.928702 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.928704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.928846 (+ 142us) server_negotiation.cc:407] Connection header received
0924 15:30:29.928932 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.928933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.928943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.929004 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:29.929172 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.929865 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.931456 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.932039 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.932102 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.932110 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.932563 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.932596 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:30:29.932662 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.928610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.5.98.195:50891)
0924 15:30:29.928783 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.928794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:29.928874 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:29.929009 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:29.929010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:29.929012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:29.929077 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.929078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.929964 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.929965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.931374 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:29.931377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.932405 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.932406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:29.932482 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.932492 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:30:29.932558 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:30:29.932594 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 15:30:29.932824 20951 catalog_manager.cc:5495] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "1394ccabc0804aebbec5a1f667c92d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } health_report { overall_health: HEALTHY } } }
I20260924 15:30:29.932948 21098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179921b3637c48ef92a5fe206cd114b6" candidate_uuid: "51983300742c49bb977204ee6b6068b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1394ccabc0804aebbec5a1f667c92d77" is_pre_election: true
I20260924 15:30:29.933193 21306 leader_election.cc:302] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51983300742c49bb977204ee6b6068b7; no voters: 1394ccabc0804aebbec5a1f667c92d77, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:30:29.933313 21443 raft_consensus.cc:2726] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 15:30:29.948438 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.946151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.5.98.193:36281)
0924 15:30:29.946230 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:29.946232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:29.946242 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:29.946277 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:29.946278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:29.946298 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:29.946366 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:29.946502 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.946952 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.947732 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:29.947911 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:29.947964 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:29.948100 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:29.948200 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:29.948261 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:29.948293 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 15:30:29.948311 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:30:29.950482 21234 raft_consensus.cc:1261] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Refusing update from remote peer 1394ccabc0804aebbec5a1f667c92d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:29.950572 21371 raft_consensus.cc:1261] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Refusing update from remote peer 1394ccabc0804aebbec5a1f667c92d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:29.950845 21446 consensus_queue.cc:1035] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:29.950986 21442 consensus_queue.cc:1035] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:29.953133 21449 mvcc.cc:205] Tried to move back new op lower bound from 7332920647474339840 to 7332920647399092224. Current Snapshot: MvccSnapshot[applied={T|T < 7332920647474339840}]
I20260924 15:30:29.953491 21451 mvcc.cc:205] Tried to move back new op lower bound from 7332920647474339840 to 7332920647399092224. Current Snapshot: MvccSnapshot[applied={T|T < 7332920647474339840}]
I20260924 15:30:29.956146 20950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:29.956246 20950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:29.956972 20950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Sending DeleteTablet for 3 replicas of tablet 179921b3637c48ef92a5fe206cd114b6
I20260924 15:30:29.957314 21350 tablet_service.cc:1514] Processing DeleteTablet for tablet 179921b3637c48ef92a5fe206cd114b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:47414
I20260924 15:30:29.957299 21214 tablet_service.cc:1514] Processing DeleteTablet for tablet 179921b3637c48ef92a5fe206cd114b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:36200
I20260924 15:30:29.957460 21078 tablet_service.cc:1514] Processing DeleteTablet for tablet 179921b3637c48ef92a5fe206cd114b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:60006
I20260924 15:30:29.957544 21459 tablet_replica.cc:330] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: stopping tablet replica
I20260924 15:30:29.957551 21460 tablet_replica.cc:330] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: stopping tablet replica
I20260924 15:30:29.957561 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21015
I20260924 15:30:29.957638 21460 raft_consensus.cc:2229] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:29.957643 21459 raft_consensus.cc:2229] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:29.957726 21460 raft_consensus.cc:2258] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:29.957739 21459 raft_consensus.cc:2258] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:29.958124 21460 ts_tablet_manager.cc:1905] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:29.959386 21459 ts_tablet_manager.cc:1905] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:29.960233 21460 ts_tablet_manager.cc:1918] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:29.960325 21460 log.cc:1196] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/179921b3637c48ef92a5fe206cd114b6
I20260924 15:30:29.960584 21460 ts_tablet_manager.cc:1939] T 179921b3637c48ef92a5fe206cd114b6 P 51983300742c49bb977204ee6b6068b7: Deleting consensus metadata
I20260924 15:30:29.960987 20937 catalog_manager.cc:4841] TS 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): tablet 179921b3637c48ef92a5fe206cd114b6 (table test-table [id=abb94493152848d486a07b7b2d6c8f9c]) successfully deleted
I20260924 15:30:29.961122 21459 ts_tablet_manager.cc:1918] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:29.961190 21459 log.cc:1196] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/179921b3637c48ef92a5fe206cd114b6
I20260924 15:30:29.961448 21459 ts_tablet_manager.cc:1939] T 179921b3637c48ef92a5fe206cd114b6 P 68e91b3e3cfa49069f3f63db85f95161: Deleting consensus metadata
I20260924 15:30:29.961851 20935 catalog_manager.cc:4841] TS 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): tablet 179921b3637c48ef92a5fe206cd114b6 (table test-table [id=abb94493152848d486a07b7b2d6c8f9c]) successfully deleted
W20260924 15:30:29.964107 20938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:36281 (error 108)
I20260924 15:30:29.964359 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21151
I20260924 15:30:29.964942 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:29.964506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.0.0.1:60018)
0924 15:30:29.964862 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:29.964904 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
W20260924 15:30:29.965121 20938 catalog_manager.cc:4584] TS 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 179921b3637c48ef92a5fe206cd114b6: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36281: connect: Connection refused (error 111)
I20260924 15:30:29.969983 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21287
I20260924 15:30:29.974825 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 20919
I20260924 15:30:49.980604 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--webserver_interface=127.5.98.254
--webserver_port=45495
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:30:50.064252 21463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:50.064473 21463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:50.064498 21463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:50.065685 21463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:30:50.065725 21463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:30:50.065739 21463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:30:50.065750 21463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:30:50.065763 21463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:50.065778 21463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:50.065788 21463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:30:50.065845 21463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:30:50.067342 21463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:50.067626 21463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:50.067898 21463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:50.077358 21469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.077358 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.077332 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:50.077353 21463 server_base.cc:1034] running on GCE node
I20260924 15:30:50.078009 21463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:50.078328 21463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:50.079499 21463 hybrid_clock.cc:648] HybridClock initialized: now 1790263850079478 us; error 40 us; skew 500 ppm
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:50.089211 21463 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:30:50.089771 21463 webserver.cc:460] Webserver started at http://127.5.98.254:45495/ using document root <none> and password file <none>
I20260924 15:30:50.090039 21463 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:50.090098 21463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:50.091648 21463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:50.092463 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.092669 21463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:50.092797 21463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "978a301706ca4e05be8ce733502a3ade"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "33272b4b403e5f386d30b2e2454c895d"
server_key_iv: "54fffaffb2762f14a5fe59b73ad4b56f"
server_key_version: "encryptionkey@0"
I20260924 15:30:50.093236 21463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:50.102452 21463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:50.104758 21463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:50.104931 21463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:50.109189 21463 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36519
I20260924 15:30:50.109257 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36519 every 8 connection(s)
I20260924 15:30:50.109679 21463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:30:50.111217 21532 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:30:50.112005 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap starting.
I20260924 15:30:50.115907 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21463
I20260924 15:30:50.116235 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:36281
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=38927
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:50.121307 21532 log.cc:824] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:50.123234 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:50.123562 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap complete.
I20260924 15:30:50.125849 21532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:50.126075 21532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Initialized, Role: FOLLOWER
I20260924 15:30:50.126248 21532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:50.126312 21532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:30:50.126363 21532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:30:50.126421 21532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:30:50.127458 21532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:50.127573 21532 leader_election.cc:302] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978a301706ca4e05be8ce733502a3ade; no voters:
I20260924 15:30:50.127815 21532 leader_election.cc:288] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:30:50.127848 21536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:30:50.127977 21536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 LEADER]: Becoming Leader. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Running, Role: LEADER
I20260924 15:30:50.128098 21532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:30:50.128132 21536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:30:50.128423 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:30:50.128608 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:50.128419 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978a301706ca4e05be8ce733502a3ade. Latest consensus state: current_term: 2 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:30:50.128993 21544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:30:50.129069 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:30:50.129917 21544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:30:50.130082 21544 catalog_manager.cc:1259] Loaded cluster ID: e7b9a436d0624dea880c39a58d1cc197
I20260924 15:30:50.130306 21544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:30:50.131570 21544 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:30:50.131835 21544 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Loaded TSK: 0
I20260924 15:30:50.142642 21544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Generated new TSK 1
I20260924 15:30:50.142752 21544 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:30:50.203011 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:50.203210 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:50.203233 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:50.204403 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:50.204444 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:50.204470 21534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:30:50.205891 21534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=38927
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:50.206126 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:50.206382 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:50.212345 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.212345 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.212352 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:50.212597 21534 server_base.cc:1034] running on GCE node
I20260924 15:30:50.212806 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:50.213061 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:50.214218 21534 hybrid_clock.cc:648] HybridClock initialized: now 1790263850214194 us; error 41 us; skew 500 ppm
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:50.216723 21534 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:30:50.217221 21534 webserver.cc:460] Webserver started at http://127.5.98.193:38927/ using document root <none> and password file <none>
I20260924 15:30:50.217442 21534 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:50.217499 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:50.218922 21534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:50.219671 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.219830 21534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:50.219919 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "1394ccabc0804aebbec5a1f667c92d77"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "8a9f2037640b3bc7f1827ba0a12cc702"
server_key_iv: "e2a31bef7326729fa0ed99614e0b2e53"
server_key_version: "encryptionkey@0"
I20260924 15:30:50.220364 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:50.238832 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:50.241413 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:50.241570 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:50.241855 21534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:50.242460 21577 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:30:50.243993 21534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:30:50.244057 21534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:30:50.244093 21534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:30:50.244961 21534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:30:50.245010 21534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:50.245086 21577 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap starting.
I20260924 15:30:50.251608 21534 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:36281
I20260924 15:30:50.251811 21686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:36281 every 8 connection(s)
I20260924 15:30:50.252172 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:30:50.256310 21577 log.cc:824] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:50.258793 21577 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:30:50.259132 21577 tablet_bootstrap.cc:492] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap complete.
I20260924 15:30:50.260339 21577 ts_tablet_manager.cc:1397] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:50.261780 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21534
I20260924 15:30:50.261710 21577 raft_consensus.cc:348] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.262071 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:40875
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=40663
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:30:50.262065 21577 raft_consensus.cc:729] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1394ccabc0804aebbec5a1f667c92d77, State: Initialized, Role: FOLLOWER
I20260924 15:30:50.262230 21577 consensus_queue.cc:260] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.262528 21577 ts_tablet_manager.cc:1428] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:30:50.266880 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.252890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:44349)
0924 15:30:50.253092 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.253139 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.253717 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.254303 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.254308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.254645 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:50.254914 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.254919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.256754 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.256755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.258017 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.258020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.258914 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.258915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.259074 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.259713 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:50.259723 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:50.261538 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:50.264959 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.264966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.264969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.265300 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.265596 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.265597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.265597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.265751 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.266264 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:50.266272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:50.266423 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.266534 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.266695 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260924 15:30:50.267023 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.253005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:44349 (local address 127.5.98.254:36519)
0924 15:30:50.253226 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.253235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.253282 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:30:50.253995 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.254006 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.254086 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.254252 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.255064 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.256624 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.258124 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.258780 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.259005 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.262707 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.262711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.262712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.262725 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.264855 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.265393 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.265396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.265396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.265511 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.265844 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.265845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.265846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.266175 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.266274 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.266624 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.266787 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260924 15:30:50.267619 21687 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:50.267740 21687 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:50.267930 21687 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:50.268802 21496 ts_manager.cc:194] Registered new tserver with Master: 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281)
I20260924 15:30:50.270124 21496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:44349
I20260924 15:30:50.271422 21687 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.193@KRBTEST.COM
I20260924 15:30:50.277554 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.269499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.0.0.1:51110)
0924 15:30:50.269672 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.269723 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.269793 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.270058 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.270064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.270391 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:50.270475 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.270477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.271311 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.271313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.272141 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.272143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.272607 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.272608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.272685 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.273195 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:50.273200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:50.274639 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:50.276386 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.276401 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.276403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.276597 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.276818 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.276819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.276819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.276878 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.277172 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:50.277179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:50.277228 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.277322 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.277454 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260924 15:30:50.277552 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.269742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.5.98.193:36281)
0924 15:30:50.269913 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.269919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.269934 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:30:50.269960 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.269961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.269974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.270055 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.270564 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.271221 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.272232 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.272523 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.272607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.274730 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.274737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.274738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.274744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.276299 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.276684 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.276687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.276687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.276748 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.276953 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.276954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.276955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.277092 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.277179 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.277371 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.277459 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260924 15:30:50.278258 21620 tablet_service.cc:1514] Processing DeleteTablet for tablet 179921b3637c48ef92a5fe206cd114b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:29 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:51110
I20260924 15:30:50.278841 21696 tablet_replica.cc:330] stopping tablet replica
I20260924 15:30:50.278957 21696 raft_consensus.cc:2229] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:50.279045 21696 raft_consensus.cc:2258] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:50.279340 21696 ts_tablet_manager.cc:1905] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:50.281628 21696 ts_tablet_manager.cc:1918] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:50.281716 21696 log.cc:1196] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal/wals/179921b3637c48ef92a5fe206cd114b6
I20260924 15:30:50.281934 21696 ts_tablet_manager.cc:1939] T 179921b3637c48ef92a5fe206cd114b6 P 1394ccabc0804aebbec5a1f667c92d77: Deleting consensus metadata
I20260924 15:30:50.282416 21483 catalog_manager.cc:4841] TS 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281): tablet 179921b3637c48ef92a5fe206cd114b6 (table test-table [id=abb94493152848d486a07b7b2d6c8f9c]) successfully deleted
W20260924 15:30:50.347766 21693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:50.347965 21693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:50.347988 21693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:50.349150 21693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:50.349191 21693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:50.349216 21693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:30:50.350654 21693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:40875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=40663
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:50.350898 21693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:50.351127 21693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:50.356307 21701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.356294 21704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.356294 21702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:50.356320 21693 server_base.cc:1034] running on GCE node
I20260924 15:30:50.356736 21693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:50.357035 21693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:50.358188 21693 hybrid_clock.cc:648] HybridClock initialized: now 1790263850358172 us; error 34 us; skew 500 ppm
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:50.360687 21693 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:30:50.361129 21693 webserver.cc:460] Webserver started at http://127.5.98.194:40663/ using document root <none> and password file <none>
I20260924 15:30:50.361361 21693 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:50.361438 21693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:50.362772 21693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:30:50.363350 21711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.363516 21693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:50.363613 21693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "68e91b3e3cfa49069f3f63db85f95161"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "893488965e31a171fa742dde62fc6fa7"
server_key_iv: "ff8dbd1fce42bf6d4218d90acc2f9903"
server_key_version: "encryptionkey@0"
I20260924 15:30:50.364022 21693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:50.387121 21693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:50.388998 21693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:50.389168 21693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:50.389410 21693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:50.389727 21693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:50.389783 21693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.389824 21693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:50.389853 21693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.395788 21693 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40875
I20260924 15:30:50.395828 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40875 every 8 connection(s)
I20260924 15:30:50.396200 21693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:30:50.397575 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21693
I20260924 15:30:50.397845 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:42847
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=38855
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:50.405562 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.396953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:32873 (local address 127.5.98.254:36519)
0924 15:30:50.397024 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.397026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.397245 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:30:50.397799 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.397800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.397816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.397874 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.398868 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.399406 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.400421 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.400637 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.400723 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.402787 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.402789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.402789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.402796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.404196 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.404612 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.404613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.404614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.404670 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.404908 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.404909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.404909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.405018 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.405087 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.405371 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.405472 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:30:50.405735 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.396942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:32873)
0924 15:30:50.397126 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.397160 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.397699 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.397955 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.397959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.398567 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:50.398759 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.398764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.399515 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.399516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.400325 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.400327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.400735 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.400736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.400865 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.401380 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:50.401388 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:50.402690 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:50.404309 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.404317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.404318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.404529 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.404740 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.404740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.404741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.404840 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.405090 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:50.405097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:50.405225 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.405323 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.405505 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260924 15:30:50.406273 21827 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:50.406355 21827 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:50.406549 21827 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:50.406986 21496 ts_manager.cc:194] Registered new tserver with Master: 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
I20260924 15:30:50.407380 21496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:32873
W20260924 15:30:50.480397 21830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:30:50.480626 21830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:30:50.480666 21830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:30:50.481839 21830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:30:50.481920 21830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:30:50.481966 21830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:30:50.483469 21830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:42847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=38855
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:30:50.483758 21830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:30:50.484014 21830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:30:50.489146 21835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:50.489203 21830 server_base.cc:1034] running on GCE node
W20260924 15:30:50.489140 21836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:30:50.489140 21838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:30:50.489596 21830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:30:50.489851 21830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:30:50.491011 21830 hybrid_clock.cc:648] HybridClock initialized: now 1790263850491011 us; error 41 us; skew 500 ppm
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:30:50.493549 21830 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:30:50.493975 21830 webserver.cc:460] Webserver started at http://127.5.98.195:38855/ using document root <none> and password file <none>
I20260924 15:30:50.494199 21830 fs_manager.cc:359] Metadata directory not provided
I20260924 15:30:50.494282 21830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:30:50.495731 21830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:30:50.496368 21845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.496531 21830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:30:50.496634 21830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "51983300742c49bb977204ee6b6068b7"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "5cad1220538475632907367959e8216b"
server_key_iv: "3bd78f43b7ae53fa3f9f4eb0a58e8ff0"
server_key_version: "encryptionkey@0"
I20260924 15:30:50.497037 21830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:30:50.505363 21830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:30:50.507589 21830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:30:50.507767 21830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:30:50.507985 21830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:30:50.508282 21830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:30:50.508335 21830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.508376 21830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:30:50.508404 21830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:30:50.514068 21830 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42847
I20260924 15:30:50.514137 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42847 every 8 connection(s)
I20260924 15:30:50.514487 21830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:30:50 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:30:50.522738 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21830
I20260924 15:30:50.523536 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.515268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:60529 (local address 127.5.98.254:36519)
0924 15:30:50.515318 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.515321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.515561 (+ 240us) server_negotiation.cc:407] Connection header received
0924 15:30:50.516088 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.516089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.516100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.516152 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.517029 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.517571 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.518534 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.518723 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.518772 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.520724 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.520725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.520725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.520733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.522110 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.522437 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.522438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.522438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.522505 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.522742 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.522743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.522743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.522887 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.522960 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.523344 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.523439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:30:50.523789 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.515269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:60529)
0924 15:30:50.515443 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.515479 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.516001 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.516204 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.516207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.516768 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:30:50.516954 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.516959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.517666 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.517667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.518463 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.518465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.518792 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.518793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.518893 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.519303 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:30:50.519311 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:30:50.520641 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:30:50.522178 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.522183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.522185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.522374 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.522575 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:30:50.522576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:30:50.522576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:30:50.522665 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:30:50.522971 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:30:50.522978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:30:50.523140 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.523296 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.523514 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260924 15:30:50.524245 21962 heartbeater.cc:347] Connected to a master server at 127.5.98.254:36519
I20260924 15:30:50.524355 21962 heartbeater.cc:464] Registering TS with master...
I20260924 15:30:50.524546 21962 heartbeater.cc:511] Master 127.5.98.254:36519 requested a full tablet report, sending...
I20260924 15:30:50.524937 21496 ts_manager.cc:194] Registered new tserver with Master: 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847)
I20260924 15:30:50.525458 21496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:60529
I20260924 15:30:50.529953 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.523157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.5.98.254:36519)
0924 15:30:50.523290 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.523292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.523397 (+ 105us) server_negotiation.cc:407] Connection header received
0924 15:30:50.523459 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.523460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.523473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.523537 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.524368 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.525092 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.526082 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.526330 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.526386 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.527513 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.527515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.527515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.527521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.528873 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.529307 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.529308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.529309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.529380 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.529541 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.529542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.529542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.529633 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.529743 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.529775 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.529849 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260924 15:30:50.530607 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:30:50.533206 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.531001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.5.98.254:36519)
0924 15:30:50.531068 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.531070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.531206 (+ 136us) server_negotiation.cc:407] Connection header received
0924 15:30:50.531238 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.531239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.531267 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.531354 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:50.531429 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.531930 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.532628 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.532823 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.532877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.533008 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:50.533045 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 15:30:50.533160 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55866: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 15:30:50.533244 21966 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55866: token expired
W20260924 15:30:50.533491 21421 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.98.254:36519)
I20260924 15:30:50.538946 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.533739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.5.98.254:36519)
0924 15:30:50.533795 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.533796 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.533804 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:30:50.533865 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.533866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.533873 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.533921 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:30:50.534320 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.534748 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.535403 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.535567 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.535614 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.536825 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:30:50.536829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:30:50.536829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:30:50.536840 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:30:50.538085 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.538376 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.538377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.538377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.538425 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:30:50.538570 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:30:50.538571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:30:50.538571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:30:50.538650 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:30:50.538715 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.538793 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.538854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:30:50.540087 21496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:30:50.541726 21496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:30:50.543985 21620 tablet_service.cc:1467] Processing CreateTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 (DEFAULT_TABLE table=test-table [id=17c33fb50d8e40d783514ced245b3fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:50.544200 21620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf68923d12644b7ea093f6ff7055ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:50.545868 21577 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap starting.
I20260924 15:30:50.546211 21577 tablet_bootstrap.cc:654] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:50.546820 21577 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: No bootstrap required, opened a new log
I20260924 15:30:50.546870 21577 ts_tablet_manager.cc:1397] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:30:50.547232 21577 raft_consensus.cc:348] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.547334 21577 raft_consensus.cc:374] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:50.547395 21577 raft_consensus.cc:729] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1394ccabc0804aebbec5a1f667c92d77, State: Initialized, Role: FOLLOWER
I20260924 15:30:50.547449 21577 consensus_queue.cc:260] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.547545 21577 ts_tablet_manager.cc:1428] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:30:50.548193 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.543722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42847 (local address 127.0.0.1:41654)
0924 15:30:50.543765 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.543775 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.543810 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.544190 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.544192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.544193 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:50.544236 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.544238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.545722 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.545723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.546911 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.546914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.547933 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.547934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.548004 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.548025 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.548078 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.548128 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 15:30:50.548528 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.543790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.5.98.195:42847)
0924 15:30:50.543975 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.543980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.543990 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:50.544030 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.544032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.544047 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.544356 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:50.544445 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.545635 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.547009 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.547870 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.547936 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.547998 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.548386 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.548443 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260924 15:30:50.549153 21894 tablet_service.cc:1467] Processing CreateTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 (DEFAULT_TABLE table=test-table [id=17c33fb50d8e40d783514ced245b3fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:50.549468 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.543964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40875 (local address 127.0.0.1:57206)
0924 15:30:50.544090 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.544100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.544152 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.546149 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.546149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.546152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:50.546197 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.546199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.547311 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.547312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.548467 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.548470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.549242 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.549243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.549309 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.549313 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.549370 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.549408 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 15:30:50.549499 21894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf68923d12644b7ea093f6ff7055ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:50.549726 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.543828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.5.98.194:40875)
0924 15:30:50.545966 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.545971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.545982 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:30:50.546022 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.546024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.546040 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.546141 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:50.546262 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.547223 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.548542 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.549136 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.549237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.549296 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.549594 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.549646 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2091,"thread_start_us":72,"threads_started":1}
I20260924 15:30:50.550328 21760 tablet_service.cc:1467] Processing CreateTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 (DEFAULT_TABLE table=test-table [id=17c33fb50d8e40d783514ced245b3fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:30:50.550643 21760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf68923d12644b7ea093f6ff7055ddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:30:50.552512 21972 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Bootstrap starting.
I20260924 15:30:50.553217 21972 tablet_bootstrap.cc:654] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:50.553642 21973 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Bootstrap starting.
I20260924 15:30:50.553838 21972 log.cc:824] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:50.554255 21973 tablet_bootstrap.cc:654] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Neither blocks nor log segments found. Creating new log.
I20260924 15:30:50.554739 21972 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: No bootstrap required, opened a new log
I20260924 15:30:50.554806 21972 ts_tablet_manager.cc:1397] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:30:50.554801 21973 log.cc:824] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Log is configured to *not* fsync() on all Append() calls
I20260924 15:30:50.555744 21973 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: No bootstrap required, opened a new log
I20260924 15:30:50.555838 21973 ts_tablet_manager.cc:1397] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:30:50.556511 21972 raft_consensus.cc:348] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.556644 21972 raft_consensus.cc:374] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:50.556665 21972 raft_consensus.cc:729] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51983300742c49bb977204ee6b6068b7, State: Initialized, Role: FOLLOWER
I20260924 15:30:50.556783 21972 consensus_queue.cc:260] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.557071 21972 ts_tablet_manager.cc:1428] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:30:50.557099 21962 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
I20260924 15:30:50.557432 21973 raft_consensus.cc:348] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.557549 21973 raft_consensus.cc:374] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:30:50.557569 21973 raft_consensus.cc:729] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e91b3e3cfa49069f3f63db85f95161, State: Initialized, Role: FOLLOWER
I20260924 15:30:50.557698 21973 consensus_queue.cc:260] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.557984 21973 ts_tablet_manager.cc:1428] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:30:50.558009 21827 heartbeater.cc:503] Master 127.5.98.254:36519 was elected leader, sending a full tablet report...
I20260924 15:30:50.625139 21977 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:30:50.625304 21977 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.625665 21977 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281)
I20260924 15:30:50.629788 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.625932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:34993 (local address 127.5.98.193:36281)
0924 15:30:50.625981 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.625983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.625993 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:30:50.626048 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.626050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.626076 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.626144 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:50.626385 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.627187 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.628755 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.629448 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.629521 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.629605 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.629669 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.629712 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:30:50.629787 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.625921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:57317 (local address 127.5.98.195:42847)
0924 15:30:50.625979 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.625981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.626189 (+ 208us) server_negotiation.cc:407] Connection header received
0924 15:30:50.626221 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.626222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.626232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.626284 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:30:50.626444 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.627186 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.628755 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.629322 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.629431 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.629442 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.629670 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.629715 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:30:50.629814 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.625867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.5.98.194:34993)
0924 15:30:50.625923 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.625936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.625981 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.626170 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.626171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.626173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:50.626257 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.626259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.627278 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.627279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.628659 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.628662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.629530 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.629531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.629594 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.629597 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.629663 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.629704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":66}
I20260924 15:30:50.629822 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.625879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42847 (local address 127.5.98.194:57317)
0924 15:30:50.626095 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.626103 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.626170 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.626304 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:30:50.626305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:30:50.626306 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:30:50.626370 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.626372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.627278 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.627279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.628659 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:30:50.628662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.629490 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.629492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:30:50.629579 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.629586 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:30:50.629649 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:30:50.629704 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":12,"thread_start_us":138,"threads_started":1}
I20260924 15:30:50.630139 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf68923d12644b7ea093f6ff7055ddd9" candidate_uuid: "68e91b3e3cfa49069f3f63db85f95161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1394ccabc0804aebbec5a1f667c92d77" is_pre_election: true
I20260924 15:30:50.630306 21640 raft_consensus.cc:2454] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68e91b3e3cfa49069f3f63db85f95161 in term 0.
I20260924 15:30:50.630558 21715 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77, 68e91b3e3cfa49069f3f63db85f95161; no voters:
I20260924 15:30:50.630544 21915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf68923d12644b7ea093f6ff7055ddd9" candidate_uuid: "68e91b3e3cfa49069f3f63db85f95161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51983300742c49bb977204ee6b6068b7" is_pre_election: true
I20260924 15:30:50.630695 21915 raft_consensus.cc:2454] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68e91b3e3cfa49069f3f63db85f95161 in term 0.
I20260924 15:30:50.630705 21977 raft_consensus.cc:2781] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:30:50.630782 21977 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:30:50.630820 21977 raft_consensus.cc:3037] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:50.631610 21977 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.631774 21977 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [CANDIDATE]: Term 1 election: Requested vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281)
I20260924 15:30:50.631935 21915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf68923d12644b7ea093f6ff7055ddd9" candidate_uuid: "68e91b3e3cfa49069f3f63db85f95161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51983300742c49bb977204ee6b6068b7"
I20260924 15:30:50.632014 21915 raft_consensus.cc:3037] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:50.632402 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf68923d12644b7ea093f6ff7055ddd9" candidate_uuid: "68e91b3e3cfa49069f3f63db85f95161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1394ccabc0804aebbec5a1f667c92d77"
I20260924 15:30:50.632493 21640 raft_consensus.cc:3037] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:30:50.633035 21915 raft_consensus.cc:2454] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68e91b3e3cfa49069f3f63db85f95161 in term 1.
I20260924 15:30:50.633288 21714 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161; no voters:
I20260924 15:30:50.633458 21977 raft_consensus.cc:2781] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:30:50.633493 21640 raft_consensus.cc:2454] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68e91b3e3cfa49069f3f63db85f95161 in term 1.
I20260924 15:30:50.633630 21977 raft_consensus.cc:686] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 LEADER]: Becoming Leader. State: Replica: 68e91b3e3cfa49069f3f63db85f95161, State: Running, Role: LEADER
I20260924 15:30:50.633725 21977 consensus_queue.cc:237] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:30:50.634619 21496 catalog_manager.cc:5495] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194). New cstate: current_term: 1 leader_uuid: "68e91b3e3cfa49069f3f63db85f95161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } health_report { overall_health: UNKNOWN } } }
I20260924 15:30:50.642954 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:30:50.640879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.5.98.194:40875)
0924 15:30:50.640936 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:30:50.640951 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:30:50.640960 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:30:50.640987 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:30:50.640988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:30:50.641003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:30:50.641067 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:30:50.641172 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.641621 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.642263 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:30:50.642437 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:30:50.642489 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:30:50.642605 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:30:50.642698 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:30:50.642756 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:30:50.642863 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:30:50.642881 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:30:50.644747 21915 raft_consensus.cc:1261] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Refusing update from remote peer 68e91b3e3cfa49069f3f63db85f95161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:50.644745 21640 raft_consensus.cc:1261] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Refusing update from remote peer 68e91b3e3cfa49069f3f63db85f95161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:30:50.645066 21977 consensus_queue.cc:1035] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:30:50.645241 21979 consensus_queue.cc:1035] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:30:50.646852 21828 tablet.cc:2369] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:30:50.647981 21984 mvcc.cc:205] Tried to move back new op lower bound from 7332920732238147584 to 7332920732196454400. Current Snapshot: MvccSnapshot[applied={T|T < 7332920732238147584 or (T in {7332920732238147584})}]
I20260924 15:30:50.648274 21983 mvcc.cc:205] Tried to move back new op lower bound from 7332920732238147584 to 7332920732196454400. Current Snapshot: MvccSnapshot[applied={T|T < 7332920732238147584 or (T in {7332920732238147584})}]
I20260924 15:30:50.649793 21495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:50.649905 21495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:30:50.650633 21495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Sending DeleteTablet for 3 replicas of tablet bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:30:50.650945 21620 tablet_service.cc:1514] Processing DeleteTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:50 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:51110
I20260924 15:30:50.650967 21894 tablet_service.cc:1514] Processing DeleteTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:50 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:41654
I20260924 15:30:50.651032 21696 tablet_replica.cc:330] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: stopping tablet replica
I20260924 15:30:50.651078 21696 raft_consensus.cc:2229] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:50.651132 21696 raft_consensus.cc:2258] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:50.651115 21760 tablet_service.cc:1514] Processing DeleteTablet for tablet bf68923d12644b7ea093f6ff7055ddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:30:50 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:57206
I20260924 15:30:50.651187 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21534
I20260924 15:30:50.652272 21992 tablet_replica.cc:330] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: stopping tablet replica
I20260924 15:30:50.652370 21992 raft_consensus.cc:2229] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:30:50.652484 21992 raft_consensus.cc:2258] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:30:50.652940 21992 ts_tablet_manager.cc:1905] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:30:50.655026 21992 ts_tablet_manager.cc:1918] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:50.655112 21992 log.cc:1196] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:30:50.655381 21992 ts_tablet_manager.cc:1939] T bf68923d12644b7ea093f6ff7055ddd9 P 68e91b3e3cfa49069f3f63db85f95161: Deleting consensus metadata
I20260924 15:30:50.655788 21480 catalog_manager.cc:4841] TS 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): tablet bf68923d12644b7ea093f6ff7055ddd9 (table test-table [id=17c33fb50d8e40d783514ced245b3fa7]) successfully deleted
I20260924 15:30:50.656335 21991 tablet_replica.cc:330] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: stopping tablet replica
I20260924 15:30:50.656441 21991 raft_consensus.cc:2229] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:30:50.656543 21991 raft_consensus.cc:2258] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 15:30:50.658068 21483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:36281 (error 108)
I20260924 15:30:50.658282 21991 ts_tablet_manager.cc:1905] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 15:30:50.658675 21970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:30:50.658447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:36281 (local address 127.0.0.1:51126)
0924 15:30:50.658490 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:30:50.658503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:30:50.658548 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:30:50.658637 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 15:30:50.658790 21483 catalog_manager.cc:4584] TS 1394ccabc0804aebbec5a1f667c92d77 (127.5.98.193:36281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf68923d12644b7ea093f6ff7055ddd9: Network error: Client connection negotiation failed: client connection to 127.5.98.193:36281: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:30:50.659547 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21693
I20260924 15:30:50.664067 21991 ts_tablet_manager.cc:1918] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:30:50.664134 21991 log.cc:1196] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:30:50.664394 21991 ts_tablet_manager.cc:1939] T bf68923d12644b7ea093f6ff7055ddd9 P 51983300742c49bb977204ee6b6068b7: Deleting consensus metadata
I20260924 15:30:50.665064 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21830
I20260924 15:30:50.665086 21482 catalog_manager.cc:4841] TS 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): tablet bf68923d12644b7ea093f6ff7055ddd9 (table test-table [id=17c33fb50d8e40d783514ced245b3fa7]) successfully deleted
I20260924 15:30:50.671252 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 21463
I20260924 15:31:10.678175 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--webserver_interface=127.5.98.254
--webserver_port=45495
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:10.762534 21994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:10.762820 21994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:10.762858 21994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:10.764060 21994 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:31:10.764117 21994 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:31:10.764142 21994 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:31:10.764170 21994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:10.764217 21994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:10.764251 21994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:10.764314 21994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:10.764361 21994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:10.765992 21994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:10.766290 21994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:10.766525 21994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:10.771924 22000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:10.771960 22002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:10.771952 21999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:10.772004 21994 server_base.cc:1034] running on GCE node
I20260924 15:31:10.772326 21994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:10.772584 21994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:10.773729 21994 hybrid_clock.cc:648] HybridClock initialized: now 1790263870773714 us; error 50 us; skew 500 ppm
Sep 24 15:31:10 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:10.781903 21994 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:10.782480 21994 webserver.cc:460] Webserver started at http://127.5.98.254:45495/ using document root <none> and password file <none>
I20260924 15:31:10.782732 21994 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:10.782783 21994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:10.784318 21994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:31:10.785063 22009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:10.785264 21994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:10.785373 21994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "978a301706ca4e05be8ce733502a3ade"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "33272b4b403e5f386d30b2e2454c895d"
server_key_iv: "54fffaffb2762f14a5fe59b73ad4b56f"
server_key_version: "encryptionkey@0"
I20260924 15:31:10.785763 21994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:10.805739 21994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:10.808085 21994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:10.808249 21994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:10.812372 21994 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36519
I20260924 15:31:10.812433 22062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36519 every 8 connection(s)
I20260924 15:31:10.812799 21994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:10.813439 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 21994
I20260924 15:31:10.813769 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:36281
--local_ip_for_outbound_sockets=127.5.98.193
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=38927
--webserver_interface=127.5.98.193
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:31:10.815003 22063 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:31:10.815872 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap starting.
I20260924 15:31:10.824666 22063 log.cc:824] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:10.827049 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:31:10.827369 22063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Bootstrap complete.
I20260924 15:31:10.829628 22063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:31:10.829865 22063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Initialized, Role: FOLLOWER
I20260924 15:31:10.830041 22063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:31:10.830108 22063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:10.830166 22063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:10.830220 22063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:31:10.831238 22063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:31:10.831357 22063 leader_election.cc:302] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 978a301706ca4e05be8ce733502a3ade; no voters:
I20260924 15:31:10.831568 22063 leader_election.cc:288] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:31:10.831609 22067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:31:10.831784 22067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [term 3 LEADER]: Becoming Leader. State: Replica: 978a301706ca4e05be8ce733502a3ade, State: Running, Role: LEADER
I20260924 15:31:10.831869 22063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:10.831878 22067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } }
I20260924 15:31:10.832329 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978a301706ca4e05be8ce733502a3ade. Latest consensus state: current_term: 3 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:31:10.832432 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:10.832365 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "978a301706ca4e05be8ce733502a3ade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "978a301706ca4e05be8ce733502a3ade" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36519 } } }
I20260924 15:31:10.832587 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:10.832653 22075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:10.833503 22075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:10.833621 22075 catalog_manager.cc:1259] Loaded cluster ID: e7b9a436d0624dea880c39a58d1cc197
I20260924 15:31:10.833673 22075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:10.834836 22075 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:10.835017 22075 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Loaded TSK: 0
I20260924 15:31:10.835088 22075 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 978a301706ca4e05be8ce733502a3ade: Loaded TSK: 1
F20260924 15:31:10.835322 22075 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdc5a36a9ad google::LogMessage::Fail() at ??:0
@ 0x7fdc5a36d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdc5a36a4de google::LogMessage::Flush() at ??:0
@ 0x7fdc5a36ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc5cad7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdc5cad4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdc622c41cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdc622c448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdc622c4594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fdc6228ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdc62294ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdc5c2380d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdc5c22d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc5d0916db start_thread at ??:0
@ 0x7fdc58e6861f clone at ??:0
W20260924 15:31:10.922037 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:10.922309 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:10.922348 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:10.924103 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:10.924161 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:10.924202 22065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:10.926471 22065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=38927
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:10.926725 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:10.926998 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:10.932301 22105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:10.932315 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:10.932327 22104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:10.932554 22065 server_base.cc:1034] running on GCE node
I20260924 15:31:10.932727 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:10.932961 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:10.934119 22065 hybrid_clock.cc:648] HybridClock initialized: now 1790263870934107 us; error 31 us; skew 500 ppm
Sep 24 15:31:10 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:10.936640 22065 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:31:10.937076 22065 webserver.cc:460] Webserver started at http://127.5.98.193:38927/ using document root <none> and password file <none>
I20260924 15:31:10.937336 22065 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:10.937382 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:10.938704 22065 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:10.939438 22114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:10.939625 22065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:10.939733 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "1394ccabc0804aebbec5a1f667c92d77"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "8a9f2037640b3bc7f1827ba0a12cc702"
server_key_iv: "e2a31bef7326729fa0ed99614e0b2e53"
server_key_version: "encryptionkey@0"
I20260924 15:31:10.940104 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:10.958917 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:10.961294 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:10.961462 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:10.961680 22065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:10.962117 22121 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:31:10.964186 22065 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:31:10.964264 22065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:31:10.964315 22065 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:31:10.964998 22065 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:31:10.965054 22065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:31:10.965129 22121 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap starting.
I20260924 15:31:10.973116 22065 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:36281
I20260924 15:31:10.973253 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:36281 every 8 connection(s)
I20260924 15:31:10.973547 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:31:10.976231 22121 log.cc:824] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:10.978876 22121 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:31:10.979214 22121 tablet_bootstrap.cc:492] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Bootstrap complete.
I20260924 15:31:10.979466 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22065
I20260924 15:31:10.979785 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:40875
--local_ip_for_outbound_sockets=127.5.98.194
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=40663
--webserver_interface=127.5.98.194
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:31:10.980499 22121 ts_tablet_manager.cc:1397] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 15:31:10.981886 22121 raft_consensus.cc:348] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:10.982236 22121 raft_consensus.cc:729] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1394ccabc0804aebbec5a1f667c92d77, State: Initialized, Role: FOLLOWER
I20260924 15:31:10.982380 22121 consensus_queue.cc:260] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:10.982650 22121 ts_tablet_manager.cc:1428] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260924 15:31:11.050276 22232 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:31:10.974347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:51395)
0924 15:31:10.974528 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:10.974573 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:10.975032 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:11.050079 (+ 75047us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
W20260924 15:31:11.050545 22230 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:36519 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:36519: Client connection negotiation failed: client connection to 127.5.98.254:36519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:31:11.050906 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.050787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:54515)
0924 15:31:11.050840 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.050886 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:31:11.051270 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.051172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:41483)
0924 15:31:11.051230 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.051254 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:31:11.051410 22230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:31:11.063618 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:11.063875 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:11.063922 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:11.065079 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:11.065140 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:11.065186 22234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:11.066722 22234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:40875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=40663
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:11.066972 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:11.067186 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:31:11.072152 22234 server_base.cc:1034] running on GCE node
W20260924 15:31:11.072187 22240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:11.072136 22243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:11.072136 22241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:11.072518 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:11.072759 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:11.073917 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790263871073900 us; error 44 us; skew 500 ppm
Sep 24 15:31:11 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:11.076301 22234 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:31:11.076731 22234 webserver.cc:460] Webserver started at http://127.5.98.194:40663/ using document root <none> and password file <none>
I20260924 15:31:11.076973 22234 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:11.077040 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:11.078330 22234 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:11.078894 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.079052 22234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.079149 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "68e91b3e3cfa49069f3f63db85f95161"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "893488965e31a171fa742dde62fc6fa7"
server_key_iv: "ff8dbd1fce42bf6d4218d90acc2f9903"
server_key_version: "encryptionkey@0"
I20260924 15:31:11.079476 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:11.088387 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:11.090233 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:11.090382 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:11.090595 22234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:11.090884 22234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:11.090934 22234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.090974 22234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:11.091001 22234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.096894 22234 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40875
I20260924 15:31:11.096932 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40875 every 8 connection(s)
I20260924 15:31:11.097275 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:31:11.098356 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.097950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:60595)
0924 15:31:11.098132 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.098207 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260924 15:31:11.098809 22366 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:36519 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:36519: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
I20260924 15:31:11.099079 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.098988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:39103)
0924 15:31:11.099031 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.099060 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 15:31:11.099387 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.099279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:35457)
0924 15:31:11.099346 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.099371 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 15:31:11.099531 22366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:31:11.104836 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22234
I20260924 15:31:11.105105 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:42847
--local_ip_for_outbound_sockets=127.5.98.195
--tserver_master_addrs=127.5.98.254:36519
--webserver_port=38855
--webserver_interface=127.5.98.195
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:11.184729 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:11.184923 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:11.184947 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:11.186076 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:11.186118 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:11.186170 22369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:11.187595 22369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:42847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=38855
--tserver_master_addrs=127.5.98.254:36519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:11.187840 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:11.188068 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:11.192966 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:11.193066 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:11.192979 22377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:11.193239 22369 server_base.cc:1034] running on GCE node
I20260924 15:31:11.193449 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:11.193671 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:11.194823 22369 hybrid_clock.cc:648] HybridClock initialized: now 1790263871194811 us; error 37 us; skew 500 ppm
Sep 24 15:31:11 dist-test-slave-4g6f krb5kdc[20906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:11.197305 22369 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:31:11.197739 22369 webserver.cc:460] Webserver started at http://127.5.98.195:38855/ using document root <none> and password file <none>
I20260924 15:31:11.197953 22369 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:11.198022 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:11.199325 22369 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:11.199920 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.200083 22369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.200181 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "51983300742c49bb977204ee6b6068b7"
format_stamp: "Formatted at 2026-09-24 15:30:29 on dist-test-slave-4g6f"
server_key: "5cad1220538475632907367959e8216b"
server_key_iv: "3bd78f43b7ae53fa3f9f4eb0a58e8ff0"
server_key_version: "encryptionkey@0"
I20260924 15:31:11.200497 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:11.221312 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:11.223579 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:11.223770 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:11.224009 22369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:11.224330 22369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:11.224381 22369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.224421 22369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:11.224450 22369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:11.230114 22369 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:42847
I20260924 15:31:11.230159 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:42847 every 8 connection(s)
I20260924 15:31:11.230485 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:11.231480 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.231126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:51379)
0924 15:31:11.231275 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.231343 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260924 15:31:11.231915 22500 heartbeater.cc:644] Failed to heartbeat to 127.5.98.254:36519 (0 consecutive failures): Network error: Failed to ping master at 127.5.98.254:36519: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
I20260924 15:31:11.232231 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.232121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:46555)
0924 15:31:11.232187 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.232213 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 15:31:11.232637 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:11.232577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:56355)
0924 15:31:11.232602 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:11.232623 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 15:31:11.232766 22500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:31:11.239830 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22369
I20260924 15:31:12.052008 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.051767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:52349)
0924 15:31:12.051909 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:12.051945 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260924 15:31:12.100258 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.100047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:37405)
0924 15:31:12.100168 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:12.100206 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 15:31:12.162552 22506 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:12.162751 22506 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:12.163131 22506 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
Sep 24 15:31:12 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.194@KRBTEST.COM
Sep 24 15:31:12 dist-test-slave-4g6f krb5kdc[20906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263870, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.195@KRBTEST.COM
I20260924 15:31:12.173254 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.163395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:34699 (local address 127.5.98.195:42847)
0924 15:31:12.163561 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:12.163574 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:12.163602 (+ 28us) server_negotiation.cc:407] Connection header received
0924 15:31:12.164209 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:12.164212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:12.164277 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:12.164425 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:12.165105 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.166007 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.166971 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.167351 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.167468 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:12.170075 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:12.170078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:12.170078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:12.170088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:12.171621 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172081 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:12.172084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:12.172085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:12.172173 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172443 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:12.172444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:12.172444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:12.172667 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:12.172760 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:12.173029 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 15:31:12.173156 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260924 15:31:12.173280 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.163431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:48073 (local address 127.5.98.194:40875)
0924 15:31:12.163586 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:12.163593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:12.163632 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:31:12.164208 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:12.164212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:12.164272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:12.164425 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:12.165094 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.165973 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.166948 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.167333 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.167452 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:12.169740 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:12.169744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:12.169745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:12.169757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:12.171555 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172080 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:12.172083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:12.172084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:12.172171 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172424 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:12.172425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:12.172425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:12.172652 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:12.172736 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:12.173025 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 15:31:12.173144 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260924 15:31:12.173280 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.163278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:42847 (local address 127.5.98.193:34699)
0924 15:31:12.163354 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:12.163365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:12.163418 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:12.164485 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:12.164488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:12.164781 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:12.164979 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:12.164985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.166107 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.166108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:12.166853 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:12.166856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.167436 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.167437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:12.167535 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:12.168334 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:12.168339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:12.169959 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:12.171760 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:12.171761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:12.171762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:12.172001 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:12.172251 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:12.172252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:12.172362 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:12.172741 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:12.172742 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:12.172863 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 15:31:12.172965 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:31:12.173153 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":51}
I20260924 15:31:12.173888 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.163278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40875 (local address 127.5.98.193:48073)
0924 15:31:12.163569 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:12.163581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:12.163645 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:12.164551 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:12.164552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:12.164806 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:12.164980 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:12.164985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.166079 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.166080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:12.166852 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:12.166854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:12.167422 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:12.167423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:12.167534 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:12.168235 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:12.168243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:12.169634 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:12.171678 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:12.171685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:12.171687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:12.171985 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:12.172245 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:12.172246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:12.172246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:12.172356 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:12.172731 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:12.172738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:12.172864 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 15:31:12.172958 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:31:12.173230 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":162,"thread_start_us":66,"threads_started":1}
W20260924 15:31:12.174501 22117 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
W20260924 15:31:12.174690 22115 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:31:12.174728 22115 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77; no voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:31:12.174821 22506 raft_consensus.cc:2726] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:31:12.233259 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:12.233060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:42257)
0924 15:31:12.233179 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:12.233224 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260924 15:31:13.052778 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:13.052521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:50125)
0924 15:31:13.052672 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:13.052713 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260924 15:31:13.100895 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:13.100706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:49775)
0924 15:31:13.100826 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:13.100859 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 15:31:13.233942 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:13.233730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:45737)
0924 15:31:13.233869 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:13.233903 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260924 15:31:13.913259 22514 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:13.913357 22514 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:13.913568 22514 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
W20260924 15:31:13.913964 22115 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
W20260924 15:31:13.914039 22117 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:31:13.914156 22117 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77; no voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:31:13.914235 22514 raft_consensus.cc:2726] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:31:14.053534 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:14.053300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:47603)
0924 15:31:14.053443 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:14.053488 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260924 15:31:14.101573 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:14.101341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:34141)
0924 15:31:14.101494 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:14.101533 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260924 15:31:14.234630 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:14.234414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:33231)
0924 15:31:14.234556 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:14.234590 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 15:31:15.054252 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:15.054040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:33129)
0924 15:31:15.054165 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:15.054204 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260924 15:31:15.102219 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:15.102030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:40285)
0924 15:31:15.102144 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:15.102179 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 15:31:15.235306 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:15.235103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:39609)
0924 15:31:15.235232 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:15.235266 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 15:31:15.480646 22521 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:15.480734 22521 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:15.480870 22521 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
W20260924 15:31:15.481288 22115 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
W20260924 15:31:15.481355 22117 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:31:15.481388 22117 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77; no voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:31:15.481468 22521 raft_consensus.cc:2726] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:31:16.054951 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:16.054738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:37183)
0924 15:31:16.054856 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:16.054889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 15:31:16.103026 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:16.102803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:44431)
0924 15:31:16.102938 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:16.102983 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260924 15:31:16.236188 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:16.235978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:54493)
0924 15:31:16.236114 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:16.236148 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 15:31:17.055595 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:17.055393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:58651)
0924 15:31:17.055514 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:17.055549 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 15:31:17.103744 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:17.103505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:33829)
0924 15:31:17.103628 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:17.103699 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 15:31:17.236995 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:17.236668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:35599)
0924 15:31:17.236912 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:17.236951 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260924 15:31:18.056306 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:18.056100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:36919)
0924 15:31:18.056222 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:18.056260 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260924 15:31:18.104352 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:18.104143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:54921)
0924 15:31:18.104273 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:18.104311 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 15:31:18.237658 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:18.237450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:55485)
0924 15:31:18.237572 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:18.237612 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260924 15:31:19.057044 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:19.056741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:59821)
0924 15:31:19.056904 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:19.056979 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260924 15:31:19.105094 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:19.104816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:43539)
0924 15:31:19.105009 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:19.105050 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260924 15:31:19.238371 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:19.238127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:54325)
0924 15:31:19.238282 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:19.238327 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260924 15:31:19.994203 22534 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:19.994319 22534 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:19.994575 22534 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
W20260924 15:31:19.995101 22117 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
W20260924 15:31:19.995177 22115 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:31:19.995210 22115 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77; no voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:31:19.995304 22534 raft_consensus.cc:2726] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:31:20.057724 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:20.057502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:36565)
0924 15:31:20.057638 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:20.057676 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260924 15:31:20.105895 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:20.105624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:44851)
0924 15:31:20.105765 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:20.105806 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 15:31:20.239068 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:20.238811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:33111)
0924 15:31:20.238965 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:20.239010 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260924 15:31:21.058539 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:21.058277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:49761)
0924 15:31:21.058417 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:21.058475 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260924 15:31:21.106572 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:21.106351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:45017)
0924 15:31:21.106490 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:21.106527 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260924 15:31:21.239766 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:21.239519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:43541)
0924 15:31:21.239641 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:21.239725 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 15:31:22.059310 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:22.059015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:42631)
0924 15:31:22.059226 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:22.059265 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":63,"threads_started":1}
I20260924 15:31:22.107364 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:22.107156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:38683)
0924 15:31:22.107287 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:22.107322 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 15:31:22.240387 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:22.240182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:51783)
0924 15:31:22.240304 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:22.240343 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 15:31:23.060057 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:23.059844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:51625)
0924 15:31:23.059976 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:23.060012 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 15:31:23.108096 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:23.107884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:51491)
0924 15:31:23.107998 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:23.108034 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260924 15:31:23.241111 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:23.240918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:36107)
0924 15:31:23.241041 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:23.241071 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 15:31:23.556308 22547 raft_consensus.cc:482] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:23.556397 22547 raft_consensus.cc:504] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51983300742c49bb977204ee6b6068b7" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 42847 } } peers { permanent_uuid: "68e91b3e3cfa49069f3f63db85f95161" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40875 } } peers { permanent_uuid: "1394ccabc0804aebbec5a1f667c92d77" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 36281 } }
I20260924 15:31:23.556620 22547 leader_election.cc:288] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847), 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875)
W20260924 15:31:23.556984 22117 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51983300742c49bb977204ee6b6068b7 (127.5.98.195:42847): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
W20260924 15:31:23.557065 22115 leader_election.cc:341] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68e91b3e3cfa49069f3f63db85f95161 (127.5.98.194:40875): Not found: Tablet not found: bf68923d12644b7ea093f6ff7055ddd9
I20260924 15:31:23.557153 22115 leader_election.cc:302] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1394ccabc0804aebbec5a1f667c92d77; no voters: 51983300742c49bb977204ee6b6068b7, 68e91b3e3cfa49069f3f63db85f95161
I20260924 15:31:23.557240 22547 raft_consensus.cc:2726] T bf68923d12644b7ea093f6ff7055ddd9 P 1394ccabc0804aebbec5a1f667c92d77 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:31:24.060683 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:24.060494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:45401)
0924 15:31:24.060603 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:24.060636 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 15:31:24.108704 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:24.108517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:56905)
0924 15:31:24.108634 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:24.108664 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260924 15:31:24.241758 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:24.241531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:53033)
0924 15:31:24.241674 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:24.241712 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260924 15:31:25.061334 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:25.061120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:52247)
0924 15:31:25.061256 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:25.061290 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 15:31:25.109318 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:25.109124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:57793)
0924 15:31:25.109246 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:25.109278 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260924 15:31:25.242578 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:25.242345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:47221)
0924 15:31:25.242504 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:25.242539 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260924 15:31:26.062037 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:26.061836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.193:53115)
0924 15:31:26.061964 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:26.061994 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 15:31:26.110203 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:26.109913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.194:39569)
0924 15:31:26.110124 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:26.110160 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260924 15:31:26.243340 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:26.243136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:36519 (local address 127.5.98.195:55621)
0924 15:31:26.243261 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:26.243297 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.254:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 15:31:26.246524 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22065
I20260924 15:31:26.251585 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22234
I20260924 15:31:26.256058 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22369
2026-09-24T15:31:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58271 ms)
[----------] 30 tests from SecurityITest (230684 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:26 dist-test-slave-4g6f krb5kdc[22560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:26 dist-test-slave-4g6f krb5kdc[22560](info): set up 2 sockets
Sep 24 15:31:26 dist-test-slave-4g6f krb5kdc[22560](info): commencing operation
krb5kdc: starting...
W20260924 15:31:27.288293 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:27.407719 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44937
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:27.488539 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:27.488727 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:27.488749 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:27.489813 22573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:27.489846 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:27.489861 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:27.489872 22573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:27.489883 22573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:27.491258 22573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:44937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:27.491436 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:27.491595 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:27.497181 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:27.497205 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.497236 22573 server_base.cc:1034] running on GCE node
W20260924 15:31:27.497205 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.497591 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:27.497879 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:27.499027 22573 hybrid_clock.cc:648] HybridClock initialized: now 1790263887499007 us; error 39 us; skew 500 ppm
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:27.506179 22573 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:27.506649 22573 webserver.cc:460] Webserver started at http://127.5.98.254:36655/ using document root <none> and password file <none>
I20260924 15:31:27.506870 22573 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:27.506915 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:27.507007 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:27.508097 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "a2b3b889270c40cd936198883ae5344e"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "99ae8aed80ae89149332d72a399880ba"
server_key_iv: "275416a481345dd075df31aeafe437df"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.508419 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "a2b3b889270c40cd936198883ae5344e"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "99ae8aed80ae89149332d72a399880ba"
server_key_iv: "275416a481345dd075df31aeafe437df"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.509708 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:27.510442 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.510629 22573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:27.510717 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "a2b3b889270c40cd936198883ae5344e"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "99ae8aed80ae89149332d72a399880ba"
server_key_iv: "275416a481345dd075df31aeafe437df"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.510776 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:27.536580 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:27.538298 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:27.538401 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:27.542214 22573 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:44937
I20260924 15:31:27.542287 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:44937 every 8 connection(s)
I20260924 15:31:27.542605 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:27.543450 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:27.545621 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: Bootstrap starting.
I20260924 15:31:27.546187 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:27.546474 22641 log.cc:824] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:27.547170 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: No bootstrap required, opened a new log
I20260924 15:31:27.549032 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } }
I20260924 15:31:27.549204 22641 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:27.549254 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b3b889270c40cd936198883ae5344e, State: Initialized, Role: FOLLOWER
I20260924 15:31:27.549460 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } }
I20260924 15:31:27.549567 22641 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:27.549613 22641 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:27.549664 22641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:27.550186 22641 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } }
I20260924 15:31:27.550324 22641 leader_election.cc:302] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2b3b889270c40cd936198883ae5344e; no voters:
I20260924 15:31:27.550511 22641 leader_election.cc:288] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:27.550557 22644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:27.550753 22644 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [term 1 LEADER]: Becoming Leader. State: Replica: a2b3b889270c40cd936198883ae5344e, State: Running, Role: LEADER
I20260924 15:31:27.550761 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:27.550853 22644 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } }
I20260924 15:31:27.551249 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2b3b889270c40cd936198883ae5344e. Latest consensus state: current_term: 1 leader_uuid: "a2b3b889270c40cd936198883ae5344e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } } }
I20260924 15:31:27.551379 22645 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2b3b889270c40cd936198883ae5344e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2b3b889270c40cd936198883ae5344e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44937 } } }
I20260924 15:31:27.551405 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:27.551471 22645 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:27.551978 22653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:27.552361 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22573
I20260924 15:31:27.552477 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:27.552538 22653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:27.552687 5515 external_mini_cluster.cc:1428] Setting key b384a0c7aa84a33eb918fd0013b2aa90
I20260924 15:31:27.554332 22653 catalog_manager.cc:1347] Generated new cluster ID: 3be88423877a4ab9a634b24b31d56eef
I20260924 15:31:27.554385 22653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:27.564232 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:27.553200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.5.98.254:44937)
0924 15:31:27.553480 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:27.553489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:27.553519 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:31:27.553637 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:27.553662 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:27.553722 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:27.553864 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:27.554637 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.556150 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.557192 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.557829 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.557941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:27.560676 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:27.560680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:27.560680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:27.560692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:27.562546 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.562971 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.562974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.562974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.563075 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.563282 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.563283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.563283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.563577 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:27.563733 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:27.563892 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 15:31:27.564015 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20260924 15:31:27.578059 22653 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:27.578572 22653 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:27.582608 22653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: Generated new TSK 0
I20260924 15:31:27.582774 22653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:31:27.614538 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:27.694208 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:27.694418 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:27.694456 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:27.695623 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:27.695705 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:27.695760 22667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:27.697194 22667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:27.697453 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:27.697677 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:27.702922 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.702989 22667 server_base.cc:1034] running on GCE node
W20260924 15:31:27.702931 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:27.702939 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.703373 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:27.703562 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:27.704699 22667 hybrid_clock.cc:648] HybridClock initialized: now 1790263887704697 us; error 41 us; skew 500 ppm
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:27.707079 22667 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:31:27.707481 22667 webserver.cc:460] Webserver started at http://127.5.98.193:40697/ using document root <none> and password file <none>
I20260924 15:31:27.707777 22667 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:27.707845 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:27.707993 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:27.708974 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "e96746ebfe4944b8bb03ea889cfaa72c"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "44e9d62d90db3b04e9a40df856b38b5a"
server_key_iv: "5ae37f24e183fd6eae2c16fc17d5c59b"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.709327 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "e96746ebfe4944b8bb03ea889cfaa72c"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "44e9d62d90db3b04e9a40df856b38b5a"
server_key_iv: "5ae37f24e183fd6eae2c16fc17d5c59b"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.710606 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:27.711309 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.711485 22667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:27.711584 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "e96746ebfe4944b8bb03ea889cfaa72c"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "44e9d62d90db3b04e9a40df856b38b5a"
server_key_iv: "5ae37f24e183fd6eae2c16fc17d5c59b"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.711694 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:27.746727 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:27.748708 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:27.748818 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:27.748979 22667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:27.749308 22667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:27.749341 22667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.749387 22667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:27.749413 22667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.755487 22667 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:41841
I20260924 15:31:27.755527 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:41841 every 8 connection(s)
I20260924 15:31:27.755894 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:31:27.759331 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22667
I20260924 15:31:27.759446 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:27.759701 5515 external_mini_cluster.cc:1428] Setting key 6ec3fc07baf1112ec38e27d27c99a170
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:27.766975 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:27.756598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:40127 (local address 127.5.98.254:44937)
0924 15:31:27.756647 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:27.756649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:27.756885 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:31:27.757482 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:27.757484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:27.757499 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:27.757555 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:27.758581 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.759116 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.760695 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.760926 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.761025 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:27.763807 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:27.763808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:27.763808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:27.763815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:27.765426 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.765939 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.765940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.765940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.766006 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.766307 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.766308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.766308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.766424 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:27.766459 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:27.766790 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 15:31:27.766875 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:31:27.767510 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:27.756569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44937 (local address 127.5.98.193:40127)
0924 15:31:27.756750 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:27.756789 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:27.757342 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:27.757630 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:27.757634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:27.758251 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:27.758461 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:27.758467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.759238 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.759239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:27.760576 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:27.760580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.761033 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.761034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:27.761187 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:27.761857 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:27.761865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:27.763718 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:27.765540 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:27.765548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:27.765550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:27.765837 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:27.766085 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:27.766086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:27.766086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:27.766215 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:27.766506 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:27.766512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:27.766619 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 15:31:27.767107 (+ 488us) client_negotiation.cc:240] Negotiation successful
0924 15:31:27.767280 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260924 15:31:27.768306 22797 heartbeater.cc:347] Connected to a master server at 127.5.98.254:44937
I20260924 15:31:27.768424 22797 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:27.768625 22797 heartbeater.cc:511] Master 127.5.98.254:44937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
I20260924 15:31:27.769181 22605 ts_manager.cc:194] Registered new tserver with Master: e96746ebfe4944b8bb03ea889cfaa72c (127.5.98.193:41841)
I20260924 15:31:27.769790 22605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:40127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:31:27.786536 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:27.866832 22802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:27.867020 22802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:27.867041 22802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:27.868158 22802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:27.868196 22802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:27.868248 22802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:27.869655 22802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:27.869876 22802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:27.870095 22802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:27.874948 22807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:27.874986 22810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.875102 22802 server_base.cc:1034] running on GCE node
W20260924 15:31:27.874992 22808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:27.875373 22802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:27.875645 22802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:27.876839 22802 hybrid_clock.cc:648] HybridClock initialized: now 1790263887876816 us; error 38 us; skew 500 ppm
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:27.879249 22802 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:31:27.879784 22802 webserver.cc:460] Webserver started at http://127.5.98.194:36421/ using document root <none> and password file <none>
I20260924 15:31:27.880036 22802 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:27.880081 22802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:27.880254 22802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:27.881264 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "4efb8ced64dd435f819d8a0f62e610f1"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "75dd8285e87eaa5c92d8ad82320244a6"
server_key_iv: "19782ad25de4e1ed5368457f88ffccb1"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.881623 22802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "4efb8ced64dd435f819d8a0f62e610f1"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "75dd8285e87eaa5c92d8ad82320244a6"
server_key_iv: "19782ad25de4e1ed5368457f88ffccb1"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.882905 22802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:31:27.883638 22817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.883852 22802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:27.883937 22802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "4efb8ced64dd435f819d8a0f62e610f1"
format_stamp: "Formatted at 2026-09-24 15:31:27 on dist-test-slave-4g6f"
server_key: "75dd8285e87eaa5c92d8ad82320244a6"
server_key_iv: "19782ad25de4e1ed5368457f88ffccb1"
server_key_version: "encryptionkey@0"
I20260924 15:31:27.883999 22802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:27.902321 22802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:27.904245 22802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:27.904356 22802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:27.904536 22802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:27.904852 22802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:27.904886 22802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.904930 22802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:27.904958 22802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:27.910633 22802 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40217
I20260924 15:31:27.910709 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40217 every 8 connection(s)
I20260924 15:31:27.911032 22802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:31:27.911502 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22802
I20260924 15:31:27.911585 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:27.911805 5515 external_mini_cluster.cc:1428] Setting key 5ff7a8afc2548076b8f287a818286e8c
Sep 24 15:31:27 dist-test-slave-4g6f krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263887, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
I20260924 15:31:27.921255 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:27.911892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:60011 (local address 127.5.98.254:44937)
0924 15:31:27.911954 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:27.911956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:27.912180 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:31:27.912687 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:27.912688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:27.912701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:27.912773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:27.913808 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.914307 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.915412 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.915733 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.915823 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:27.918240 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:27.918242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:27.918242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:27.918249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:27.919746 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.920224 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.920225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.920226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.920284 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:27.920556 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:27.920557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:27.920557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:27.920670 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:27.920743 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:27.921082 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 15:31:27.921165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:31:27.921438 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:27.911857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44937 (local address 127.5.98.194:60011)
0924 15:31:27.912065 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:27.912106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:27.912592 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:27.912825 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:27.912828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:27.913463 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:27.913704 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:27.913710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.914403 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.914405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:27.915299 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:27.915302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:27.915834 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:27.915835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:27.915967 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:27.916481 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:27.916489 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:27.918097 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:27.919846 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:27.919853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:27.919855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:27.920135 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:27.920367 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:27.920368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:27.920368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:27.920480 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:27.920747 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:27.920753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:27.920923 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 15:31:27.921048 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 15:31:27.921201 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260924 15:31:27.921959 22932 heartbeater.cc:347] Connected to a master server at 127.5.98.254:44937
I20260924 15:31:27.922058 22932 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:27.922295 22932 heartbeater.cc:511] Master 127.5.98.254:44937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
I20260924 15:31:27.922709 22605 ts_manager.cc:194] Registered new tserver with Master: 4efb8ced64dd435f819d8a0f62e610f1 (127.5.98.194:40217)
I20260924 15:31:27.923082 22605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:60011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:31:27.936715 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:42485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:28.016700 22937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:28.016887 22937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:28.016911 22937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:28.018064 22937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:28.018105 22937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:28.018131 22937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:28.019568 22937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:42485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:28.019814 22937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:28.020058 22937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:28.025023 22943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.025038 22942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.025110 22937 server_base.cc:1034] running on GCE node
W20260924 15:31:28.025038 22945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.025388 22937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:28.025585 22937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:28.026718 22937 hybrid_clock.cc:648] HybridClock initialized: now 1790263888026706 us; error 27 us; skew 500 ppm
Sep 24 15:31:28 dist-test-slave-4g6f krb5kdc[22560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263888, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:28.029117 22937 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:31:28.029537 22937 webserver.cc:460] Webserver started at http://127.5.98.195:45407/ using document root <none> and password file <none>
I20260924 15:31:28.029769 22937 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:28.029838 22937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:28.029990 22937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:28.031037 22937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "941b79ade0b54499aacced47455800a5"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "5cf396db95f25ea11db5c083ab52a24b"
server_key_iv: "9b51740fc6d7cdc62550ab2b20c141de"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.031394 22937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "941b79ade0b54499aacced47455800a5"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "5cf396db95f25ea11db5c083ab52a24b"
server_key_iv: "9b51740fc6d7cdc62550ab2b20c141de"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.032760 22937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:28.033511 22952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.033689 22937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:28.033777 22937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "941b79ade0b54499aacced47455800a5"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "5cf396db95f25ea11db5c083ab52a24b"
server_key_iv: "9b51740fc6d7cdc62550ab2b20c141de"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.033859 22937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:28.048404 22937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:28.050118 22937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:28.050273 22937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:28.050500 22937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:28.050870 22937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:28.050941 22937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.050976 22937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:28.051009 22937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.057106 22937 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46143
I20260924 15:31:28.057148 23066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46143 every 8 connection(s)
I20260924 15:31:28.057494 22937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:28.060887 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 22937
I20260924 15:31:28.061010 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:28.061189 5515 external_mini_cluster.cc:1428] Setting key 76d9bcf1bfd8748b379feaa981788861
Sep 24 15:31:28 dist-test-slave-4g6f krb5kdc[22560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263888, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:28.066525 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.058221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:33397 (local address 127.5.98.254:44937)
0924 15:31:28.058264 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.058266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.058465 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:31:28.058920 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.058920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.058930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.058987 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.059912 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.060407 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.061552 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.061851 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.061929 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.064027 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.064029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.064029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:28.064036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.065396 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:28.065752 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:28.065753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:28.065753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:28.065803 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:28.066014 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:28.066014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:28.066014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:28.066110 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.066159 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.066400 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.066464 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:31:28.066697 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.058184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44937 (local address 127.5.98.195:33397)
0924 15:31:28.058340 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.058380 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.058838 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.059038 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.059041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.059573 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:28.059828 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.059833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.060496 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.060496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.061439 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.061441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.061946 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.061946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.062116 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.062573 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:28.062582 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:28.063931 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:28.065479 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:28.065484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:28.065486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:28.065688 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:28.065863 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:28.065864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:28.065864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:28.065955 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:28.066167 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:28.066172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:28.066277 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.066355 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.066479 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 15:31:28.067171 23067 heartbeater.cc:347] Connected to a master server at 127.5.98.254:44937
I20260924 15:31:28.067250 23067 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:28.067438 23067 heartbeater.cc:511] Master 127.5.98.254:44937 requested a full tablet report, sending...
I20260924 15:31:28.067883 22605 ts_manager.cc:194] Registered new tserver with Master: 941b79ade0b54499aacced47455800a5 (127.5.98.195:46143)
I20260924 15:31:28.068249 22605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:33397
I20260924 15:31:28.072736 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:28.079569 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.074012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.5.98.254:44937)
0924 15:31:28.074066 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.074068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.074154 (+ 86us) server_negotiation.cc:407] Connection header received
0924 15:31:28.074196 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.074197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.074204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.074248 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.074750 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.075203 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.075942 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.076115 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.076166 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.077163 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.077164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.077164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:28.077170 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.078602 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:28.078891 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:28.078892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:28.078892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:28.078950 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:28.079093 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:28.079093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:28.079093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:28.079208 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.079275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.079362 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.079441 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:31:28.080986 22605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:28.082504 22605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:28.088949 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.084747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40217 (local address 127.0.0.1:39860)
0924 15:31:28.084912 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.084949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.085011 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.085346 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.085349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.085351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.085454 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.085456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086874 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.086875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.087958 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.087961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.088750 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.088751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.088806 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.088816 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.088865 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.088900 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260924 15:31:28.089005 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.084872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.5.98.194:40217)
0924 15:31:28.085141 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.085145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.085156 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:31:28.085199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.085201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.085215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.085333 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.085547 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086746 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.088042 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.088669 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.088744 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.088796 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.088890 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.088929 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260924 15:31:28.089411 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.084747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:41841 (local address 127.0.0.1:33362)
0924 15:31:28.085056 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.085252 (+ 196us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.085317 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.085767 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.085768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.085770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.085818 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.085819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086909 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.086910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.088247 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.088250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.089172 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.089173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.089235 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.089240 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.089291 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.089330 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":95,"thread_start_us":43,"threads_started":1}
I20260924 15:31:28.089687 22866 tablet_service.cc:1467] Processing CreateTablet for tablet 739df283631e4ea7acdc52e147e99acb (DEFAULT_TABLE table=test-table [id=dd43625b45f448e2b30ec3db170f7f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.089905 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.084872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.5.98.193:41841)
0924 15:31:28.085066 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.085070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.085426 (+ 356us) server_negotiation.cc:407] Connection header received
0924 15:31:28.085578 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.085581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.085599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.085683 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.085892 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086763 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.088335 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.089088 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.089170 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.089228 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.089519 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.089825 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260924 15:31:28.090056 22866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739df283631e4ea7acdc52e147e99acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.090431 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.085108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.5.98.195:46143)
0924 15:31:28.085260 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.085264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.085271 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:31:28.085319 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.085322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.085336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.085539 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.085605 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086719 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.089262 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.090088 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.090173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.090229 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.090348 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.090377 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 15:31:28.090456 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.084926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46143 (local address 127.0.0.1:47786)
0924 15:31:28.085097 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.085106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.085137 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.085435 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.085436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.085438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.085467 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.085468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.086830 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.086831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.088246 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.088250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.090207 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.090208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.090269 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.090273 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.090330 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.090366 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":87,"thread_start_us":76,"threads_started":1}
I20260924 15:31:28.091084 23001 tablet_service.cc:1467] Processing CreateTablet for tablet 739df283631e4ea7acdc52e147e99acb (DEFAULT_TABLE table=test-table [id=dd43625b45f448e2b30ec3db170f7f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.091375 23001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739df283631e4ea7acdc52e147e99acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.092330 22731 tablet_service.cc:1467] Processing CreateTablet for tablet 739df283631e4ea7acdc52e147e99acb (DEFAULT_TABLE table=test-table [id=dd43625b45f448e2b30ec3db170f7f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.092666 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739df283631e4ea7acdc52e147e99acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.093050 23084 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Bootstrap starting.
I20260924 15:31:28.093643 23084 tablet_bootstrap.cc:654] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.094058 23084 log.cc:824] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.094532 23086 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Bootstrap starting.
I20260924 15:31:28.094780 23084 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: No bootstrap required, opened a new log
I20260924 15:31:28.094849 23084 ts_tablet_manager.cc:1397] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:28.095201 23086 tablet_bootstrap.cc:654] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.095623 23087 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: Bootstrap starting.
I20260924 15:31:28.095984 23086 log.cc:824] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.096371 23087 tablet_bootstrap.cc:654] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.096803 23086 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: No bootstrap required, opened a new log
I20260924 15:31:28.096875 23086 ts_tablet_manager.cc:1397] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:31:28.096896 23087 log.cc:824] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.097033 23084 raft_consensus.cc:348] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.097210 23084 raft_consensus.cc:374] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.097245 23084 raft_consensus.cc:729] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efb8ced64dd435f819d8a0f62e610f1, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.097374 23084 consensus_queue.cc:260] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.097659 23087 tablet_bootstrap.cc:492] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: No bootstrap required, opened a new log
I20260924 15:31:28.097667 23084 ts_tablet_manager.cc:1428] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:28.097687 22932 heartbeater.cc:503] Master 127.5.98.254:44937 was elected leader, sending a full tablet report...
I20260924 15:31:28.097723 23087 ts_tablet_manager.cc:1397] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:31:28.098826 23086 raft_consensus.cc:348] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.099022 23086 raft_consensus.cc:374] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.099071 23086 raft_consensus.cc:729] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941b79ade0b54499aacced47455800a5, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.099185 23086 consensus_queue.cc:260] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.099459 23086 ts_tablet_manager.cc:1428] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:31:28.099468 23067 heartbeater.cc:503] Master 127.5.98.254:44937 was elected leader, sending a full tablet report...
I20260924 15:31:28.099730 23087 raft_consensus.cc:348] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.099917 23087 raft_consensus.cc:374] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.099965 23087 raft_consensus.cc:729] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96746ebfe4944b8bb03ea889cfaa72c, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.100106 23087 consensus_queue.cc:260] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.100366 23087 ts_tablet_manager.cc:1428] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:28.100428 22797 heartbeater.cc:503] Master 127.5.98.254:44937 was elected leader, sending a full tablet report...
W20260924 15:31:28.161590 22933 tablet.cc:2369] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:28.197770 23092 raft_consensus.cc:482] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:28.197906 23092 raft_consensus.cc:504] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.198295 23092 leader_election.cc:288] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4efb8ced64dd435f819d8a0f62e610f1 (127.5.98.194:40217), 941b79ade0b54499aacced47455800a5 (127.5.98.195:46143)
I20260924 15:31:28.201557 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.198512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40217 (local address 127.5.98.193:37575)
0924 15:31:28.198607 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.198622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.198681 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.198853 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.198855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.198857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.198952 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.198954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.199869 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.199870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.200660 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.200662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.201297 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.201298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.201380 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.201385 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.201438 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.201492 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 15:31:28.201639 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.198599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:37575 (local address 127.5.98.194:40217)
0924 15:31:28.198651 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.198653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.198685 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:31:28.198765 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.198766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.198777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.198827 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.199052 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.199793 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.200756 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.201214 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.201271 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.201278 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.201470 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.201529 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:31:28.201753 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.198658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46143 (local address 127.5.98.193:41511)
0924 15:31:28.198834 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.198843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.198895 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.199052 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.199053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.199055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.199128 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.199129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.200017 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.200018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.200843 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.200846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.201474 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.201475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.201564 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.201569 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.201629 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.201669 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260924 15:31:28.201778 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.198730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:41511 (local address 127.5.98.195:46143)
0924 15:31:28.198794 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.198796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.198938 (+ 142us) server_negotiation.cc:407] Connection header received
0924 15:31:28.198969 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.198970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.198980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.199033 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.199216 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.199929 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.200940 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.201373 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.201448 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.201455 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.201656 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.201709 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:31:28.201933 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739df283631e4ea7acdc52e147e99acb" candidate_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb8ced64dd435f819d8a0f62e610f1" is_pre_election: true
I20260924 15:31:28.202076 22886 raft_consensus.cc:2454] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96746ebfe4944b8bb03ea889cfaa72c in term 0.
I20260924 15:31:28.202179 23021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739df283631e4ea7acdc52e147e99acb" candidate_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941b79ade0b54499aacced47455800a5" is_pre_election: true
I20260924 15:31:28.202303 22684 leader_election.cc:302] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4efb8ced64dd435f819d8a0f62e610f1, e96746ebfe4944b8bb03ea889cfaa72c; no voters:
I20260924 15:31:28.202323 23021 raft_consensus.cc:2454] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96746ebfe4944b8bb03ea889cfaa72c in term 0.
I20260924 15:31:28.202428 23092 raft_consensus.cc:2781] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:28.202477 23092 raft_consensus.cc:482] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:28.202500 23092 raft_consensus.cc:3037] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.203174 23092 raft_consensus.cc:504] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.203284 23092 leader_election.cc:288] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [CANDIDATE]: Term 1 election: Requested vote from peers 4efb8ced64dd435f819d8a0f62e610f1 (127.5.98.194:40217), 941b79ade0b54499aacced47455800a5 (127.5.98.195:46143)
I20260924 15:31:28.203518 23021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739df283631e4ea7acdc52e147e99acb" candidate_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941b79ade0b54499aacced47455800a5"
I20260924 15:31:28.203518 22886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739df283631e4ea7acdc52e147e99acb" candidate_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb8ced64dd435f819d8a0f62e610f1"
I20260924 15:31:28.203603 22886 raft_consensus.cc:3037] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.203603 23021 raft_consensus.cc:3037] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.204509 22886 raft_consensus.cc:2454] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96746ebfe4944b8bb03ea889cfaa72c in term 1.
I20260924 15:31:28.204509 23021 raft_consensus.cc:2454] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96746ebfe4944b8bb03ea889cfaa72c in term 1.
I20260924 15:31:28.204700 22684 leader_election.cc:302] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4efb8ced64dd435f819d8a0f62e610f1, e96746ebfe4944b8bb03ea889cfaa72c; no voters:
I20260924 15:31:28.204797 23092 raft_consensus.cc:2781] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:28.204963 23092 raft_consensus.cc:686] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [term 1 LEADER]: Becoming Leader. State: Replica: e96746ebfe4944b8bb03ea889cfaa72c, State: Running, Role: LEADER
I20260924 15:31:28.205060 23092 consensus_queue.cc:237] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } }
I20260924 15:31:28.205924 22604 catalog_manager.cc:5495] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c reported cstate change: term changed from 0 to 1, leader changed from <none> to e96746ebfe4944b8bb03ea889cfaa72c (127.5.98.193). New cstate: current_term: 1 leader_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e96746ebfe4944b8bb03ea889cfaa72c" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 41841 } health_report { overall_health: HEALTHY } } }
I20260924 15:31:28.233572 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.231028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.5.98.193:41841)
0924 15:31:28.231104 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.231106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.231115 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:31:28.231160 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.231161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.231181 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.231260 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:28.231405 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.231942 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.232746 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.232945 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.233021 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.233200 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:28.233305 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:28.233396 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.233437 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.233458 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:31:28.235424 22886 raft_consensus.cc:1261] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 1 FOLLOWER]: Refusing update from remote peer e96746ebfe4944b8bb03ea889cfaa72c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:28.235445 23021 raft_consensus.cc:1261] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 1 FOLLOWER]: Refusing update from remote peer e96746ebfe4944b8bb03ea889cfaa72c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:28.235813 23092 consensus_queue.cc:1035] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4efb8ced64dd435f819d8a0f62e610f1" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:28.235977 23094 consensus_queue.cc:1035] T 739df283631e4ea7acdc52e147e99acb P e96746ebfe4944b8bb03ea889cfaa72c [LEADER]: Connected to new peer: Peer: permanent_uuid: "941b79ade0b54499aacced47455800a5" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:28.238574 23098 mvcc.cc:205] Tried to move back new op lower bound from 7332920886209425408 to 7332920886088630272. Current Snapshot: MvccSnapshot[applied={T|T < 7332920886209425408}]
I20260924 15:31:28.238694 23099 mvcc.cc:205] Tried to move back new op lower bound from 7332920886209425408 to 7332920886088630272. Current Snapshot: MvccSnapshot[applied={T|T < 7332920886209425408}]
I20260924 15:31:28.240928 22603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:28.241029 22603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:28.241825 22603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2b3b889270c40cd936198883ae5344e: Sending DeleteTablet for 3 replicas of tablet 739df283631e4ea7acdc52e147e99acb
I20260924 15:31:28.242169 22731 tablet_service.cc:1514] Processing DeleteTablet for tablet 739df283631e4ea7acdc52e147e99acb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:33362
I20260924 15:31:28.242198 22866 tablet_service.cc:1514] Processing DeleteTablet for tablet 739df283631e4ea7acdc52e147e99acb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:39860
I20260924 15:31:28.242199 23001 tablet_service.cc:1514] Processing DeleteTablet for tablet 739df283631e4ea7acdc52e147e99acb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:47786
I20260924 15:31:28.242494 23109 tablet_replica.cc:330] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: stopping tablet replica
I20260924 15:31:28.242589 23109 raft_consensus.cc:2229] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:28.242691 23109 raft_consensus.cc:2258] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:28.242715 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22667
I20260924 15:31:28.243283 23110 tablet_replica.cc:330] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: stopping tablet replica
I20260924 15:31:28.243402 23110 raft_consensus.cc:2229] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:28.243507 23110 raft_consensus.cc:2258] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:28.243943 23110 ts_tablet_manager.cc:1905] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:28.244300 23109 ts_tablet_manager.cc:1905] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:28.246001 23109 ts_tablet_manager.cc:1918] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:28.246057 23110 ts_tablet_manager.cc:1918] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:28.246090 23109 log.cc:1196] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/739df283631e4ea7acdc52e147e99acb
I20260924 15:31:28.246127 23110 log.cc:1196] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/739df283631e4ea7acdc52e147e99acb
I20260924 15:31:28.246353 23109 ts_tablet_manager.cc:1939] T 739df283631e4ea7acdc52e147e99acb P 941b79ade0b54499aacced47455800a5: Deleting consensus metadata
I20260924 15:31:28.246417 23110 ts_tablet_manager.cc:1939] T 739df283631e4ea7acdc52e147e99acb P 4efb8ced64dd435f819d8a0f62e610f1: Deleting consensus metadata
I20260924 15:31:28.246814 22590 catalog_manager.cc:4841] TS 4efb8ced64dd435f819d8a0f62e610f1 (127.5.98.194:40217): tablet 739df283631e4ea7acdc52e147e99acb (table test-table [id=dd43625b45f448e2b30ec3db170f7f3d]) successfully deleted
I20260924 15:31:28.246920 22590 catalog_manager.cc:4841] TS 941b79ade0b54499aacced47455800a5 (127.5.98.195:46143): tablet 739df283631e4ea7acdc52e147e99acb (table test-table [id=dd43625b45f448e2b30ec3db170f7f3d]) successfully deleted
W20260924 15:31:28.250295 22591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:41841 (error 108)
I20260924 15:31:28.250559 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22802
I20260924 15:31:28.250813 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.250694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:41841 (local address 127.0.0.1:33376)
0924 15:31:28.250742 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.250775 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:31:28.250916 22591 catalog_manager.cc:4584] TS e96746ebfe4944b8bb03ea889cfaa72c (127.5.98.193:41841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 739df283631e4ea7acdc52e147e99acb: Network error: Client connection negotiation failed: client connection to 127.5.98.193:41841: connect: Connection refused (error 111)
I20260924 15:31:28.255854 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22937
I20260924 15:31:28.260310 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 22573
2026-09-24T15:31:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2004 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T15:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:28Z Disabled control of system clock
I20260924 15:31:28.287756 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43991
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:28.366448 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:28.366643 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:28.366662 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:28.367722 23115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:28.367754 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:28.367766 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:28.367779 23115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:28.367789 23115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:28.369051 23115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:28.369231 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:28.369441 23115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:28.374258 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.374274 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.374302 23115 server_base.cc:1034] running on GCE node
W20260924 15:31:28.374275 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.374739 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:28.375085 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:28.376267 23115 hybrid_clock.cc:648] HybridClock initialized: now 1790263888376234 us; error 28 us; skew 500 ppm
I20260924 15:31:28.377542 23115 webserver.cc:460] Webserver started at http://127.5.98.254:34989/ using document root <none> and password file <none>
I20260924 15:31:28.377768 23115 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:28.377844 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:28.377986 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:28.379024 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "483d4cd5a271a72650cf1d1b07bc82eb"
server_key_iv: "0b2d4d99cf85d122823fbb508a4e6f65"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.379400 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "483d4cd5a271a72650cf1d1b07bc82eb"
server_key_iv: "0b2d4d99cf85d122823fbb508a4e6f65"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.380716 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:28.381489 23129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.381669 23115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:28.381768 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "483d4cd5a271a72650cf1d1b07bc82eb"
server_key_iv: "0b2d4d99cf85d122823fbb508a4e6f65"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.381852 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:28.396682 23115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:28.396963 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:28.397112 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:28.401104 23115 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43991
I20260924 15:31:28.401136 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43991 every 8 connection(s)
I20260924 15:31:28.401460 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:28.401768 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23115
I20260924 15:31:28.401880 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:28.402033 5515 external_mini_cluster.cc:1428] Setting key 621766ff885b8d0c7ae537312d96a8c1
I20260924 15:31:28.402622 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.405802 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: Bootstrap starting.
I20260924 15:31:28.406399 23182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.406735 23182 log.cc:824] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.407564 23182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: No bootstrap required, opened a new log
I20260924 15:31:28.407887 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.402642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.5.98.254:43991)
0924 15:31:28.402834 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.402843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.402875 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:31:28.403368 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.403375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.403434 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.403596 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.403868 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.404840 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.405929 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.406347 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.406433 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.407161 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.407166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.407166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:28.407394 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.407426 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:28.407437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.407489 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.407625 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.407726 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 15:31:28.409564 23182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } }
I20260924 15:31:28.409693 23182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.409715 23182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5941950d7e2a4f7ca6bf7b6aba746f8d, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.409838 23182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } }
I20260924 15:31:28.409911 23182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:28.409968 23182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:28.410017 23182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.410516 23182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } }
I20260924 15:31:28.410601 23182 leader_election.cc:302] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5941950d7e2a4f7ca6bf7b6aba746f8d; no voters:
I20260924 15:31:28.410804 23182 leader_election.cc:288] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:28.410912 23187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:28.411072 23187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [term 1 LEADER]: Becoming Leader. State: Replica: 5941950d7e2a4f7ca6bf7b6aba746f8d, State: Running, Role: LEADER
I20260924 15:31:28.411113 23182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:28.411165 23187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } }
I20260924 15:31:28.411477 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } } }
I20260924 15:31:28.411492 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5941950d7e2a4f7ca6bf7b6aba746f8d. Latest consensus state: current_term: 1 leader_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5941950d7e2a4f7ca6bf7b6aba746f8d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43991 } } }
I20260924 15:31:28.411609 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:28.411839 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:28.411967 23196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:28.412483 23196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:28.413975 23196 catalog_manager.cc:1347] Generated new cluster ID: 0aa09783223f4e5fb281a37fb6c83246
I20260924 15:31:28.414036 23196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:28.452046 23196 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:28.452603 23196 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:28.457276 23196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: Generated new TSK 0
I20260924 15:31:28.457469 23196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:28.468743 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--builtin_ntp_servers=127.5.98.212:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:28.549186 23207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:28.549424 23207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:28.549458 23207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:28.550587 23207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:28.550648 23207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:28.550688 23207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:28.552055 23207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:28.552316 23207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:28.552606 23207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:28.557605 23215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.557688 23207 server_base.cc:1034] running on GCE node
W20260924 15:31:28.557592 23213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.557622 23212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.558017 23207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:28.558240 23207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:28.559409 23207 hybrid_clock.cc:648] HybridClock initialized: now 1790263888559390 us; error 50 us; skew 500 ppm
I20260924 15:31:28.560588 23207 webserver.cc:460] Webserver started at http://127.5.98.193:33343/ using document root <none> and password file <none>
I20260924 15:31:28.560827 23207 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:28.560894 23207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:28.561065 23207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:28.562063 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "5986dafcf7a6428da337854c48ce5ae1"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "489dba704f19405845350958db0b88a3"
server_key_iv: "63ccbf7aa5b0e2e42c97881b9a1aea7d"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.562424 23207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "5986dafcf7a6428da337854c48ce5ae1"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "489dba704f19405845350958db0b88a3"
server_key_iv: "63ccbf7aa5b0e2e42c97881b9a1aea7d"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.563710 23207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:28.564445 23221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.564625 23207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:28.564719 23207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "5986dafcf7a6428da337854c48ce5ae1"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "489dba704f19405845350958db0b88a3"
server_key_iv: "63ccbf7aa5b0e2e42c97881b9a1aea7d"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.564805 23207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:28.596006 23207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:28.596325 23207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:28.596494 23207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:28.596730 23207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:28.597064 23207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:28.597118 23207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.597159 23207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:28.597188 23207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.602756 23207 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:46119
I20260924 15:31:28.602830 23335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:46119 every 8 connection(s)
I20260924 15:31:28.603176 23207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:31:28.603843 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23207
I20260924 15:31:28.603945 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:28.604153 5515 external_mini_cluster.cc:1428] Setting key 62b7905a65336a726f1f2372f121a289
I20260924 15:31:28.605180 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--builtin_ntp_servers=127.5.98.212:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:31:28.609608 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.603959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:41049 (local address 127.5.98.254:43991)
0924 15:31:28.604007 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.604010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.604284 (+ 274us) server_negotiation.cc:407] Connection header received
0924 15:31:28.604877 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.604878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.604891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.604991 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.605789 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.606580 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.607973 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.608320 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.608398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.609264 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.609265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.609266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:28.609272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.609292 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:28.609297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.609364 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.609444 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.609489 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:31:28.609704 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.603978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43991 (local address 127.5.98.193:41049)
0924 15:31:28.604157 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.604198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.604761 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.605064 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.605067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.605075 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:28.605689 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.605694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.606693 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.606695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.607885 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.607888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.608406 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.608407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.608535 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.609118 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:28.609118 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:28.609146 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:28.609167 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:28.609370 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:28.609374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:28.609376 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.609446 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.609523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 15:31:28.610466 23336 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43991
I20260924 15:31:28.610562 23336 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:28.610774 23336 heartbeater.cc:511] Master 127.5.98.254:43991 requested a full tablet report, sending...
I20260924 15:31:28.611328 23146 ts_manager.cc:194] Registered new tserver with Master: 5986dafcf7a6428da337854c48ce5ae1 (127.5.98.193:46119)
I20260924 15:31:28.612058 23146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.193:41049
W20260924 15:31:28.686640 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:28.686825 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:28.686846 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:28.687911 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:28.687943 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:28.687968 23339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:28.689217 23339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:28.689390 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:28.689618 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:28.694449 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.694514 23339 server_base.cc:1034] running on GCE node
W20260924 15:31:28.694479 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.694487 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.694831 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:28.695097 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:28.696264 23339 hybrid_clock.cc:648] HybridClock initialized: now 1790263888696243 us; error 34 us; skew 500 ppm
I20260924 15:31:28.697505 23339 webserver.cc:460] Webserver started at http://127.5.98.194:39735/ using document root <none> and password file <none>
I20260924 15:31:28.697743 23339 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:28.697813 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:28.697991 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:28.698989 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "bf1d63095084489d93536b25aaf7b1d8"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "6d7f0974cff746b0f10c217e15cae71b"
server_key_iv: "3909fe47d40bea5fd80ce5d0603e1479"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.699344 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "bf1d63095084489d93536b25aaf7b1d8"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "6d7f0974cff746b0f10c217e15cae71b"
server_key_iv: "3909fe47d40bea5fd80ce5d0603e1479"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.700937 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:28.701788 23353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.701980 23339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:28.702083 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "bf1d63095084489d93536b25aaf7b1d8"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "6d7f0974cff746b0f10c217e15cae71b"
server_key_iv: "3909fe47d40bea5fd80ce5d0603e1479"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.702168 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:28.729677 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:28.730007 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:28.730175 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:28.730417 23339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:28.730746 23339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:28.730798 23339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.730832 23339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:28.730861 23339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.736464 23339 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:37169
I20260924 15:31:28.736512 23467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:37169 every 8 connection(s)
I20260924 15:31:28.736827 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:31:28.740644 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23339
I20260924 15:31:28.740742 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:28.740962 5515 external_mini_cluster.cc:1428] Setting key 4755235ee5dd6c9adb260b543fe0cd31
I20260924 15:31:28.742050 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--builtin_ntp_servers=127.5.98.212:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:31:28.742178 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.737496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:42727 (local address 127.5.98.254:43991)
0924 15:31:28.737581 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.737583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.737805 (+ 222us) server_negotiation.cc:407] Connection header received
0924 15:31:28.738266 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.738267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.738277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.738326 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.739018 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.739525 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.740587 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.740900 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.740975 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.741913 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.741914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.741915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:28.741920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.741936 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:28.741940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.741994 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.742074 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.742100 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:31:28.742393 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.737475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43991 (local address 127.5.98.194:42727)
0924 15:31:28.737682 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.737716 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.738186 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.738426 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.738430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.738438 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:28.738939 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.738944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.739627 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.739628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.740496 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.740499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.741016 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.741018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.741158 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.741753 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:28.741753 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:28.741782 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:28.741813 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:28.741997 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:28.742002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:28.742006 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.742076 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.742175 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260924 15:31:28.742905 23468 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43991
I20260924 15:31:28.743006 23468 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:28.743201 23468 heartbeater.cc:511] Master 127.5.98.254:43991 requested a full tablet report, sending...
I20260924 15:31:28.743621 23146 ts_manager.cc:194] Registered new tserver with Master: bf1d63095084489d93536b25aaf7b1d8 (127.5.98.194:37169)
I20260924 15:31:28.744155 23146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.194:42727
W20260924 15:31:28.823966 23471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:28.824152 23471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:28.824172 23471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:28.825246 23471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:28.825278 23471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:28.825320 23471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:28.826579 23471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:28.826767 23471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:28.827008 23471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:28.831987 23477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.832016 23479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:28.831987 23476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:28.832280 23471 server_base.cc:1034] running on GCE node
I20260924 15:31:28.832504 23471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:28.832705 23471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:28.833844 23471 hybrid_clock.cc:648] HybridClock initialized: now 1790263888833830 us; error 30 us; skew 500 ppm
I20260924 15:31:28.835119 23471 webserver.cc:460] Webserver started at http://127.5.98.195:33925/ using document root <none> and password file <none>
I20260924 15:31:28.835381 23471 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:28.835429 23471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:28.835556 23471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:28.836608 23471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "601946ce64f74459b92aaaa38b9ebd8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "2a249ac2b5d7980e778494d4e46b1fdb"
server_key_iv: "25455bde0075d5b05afd1a0f72aafb17"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.836962 23471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "601946ce64f74459b92aaaa38b9ebd8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "2a249ac2b5d7980e778494d4e46b1fdb"
server_key_iv: "25455bde0075d5b05afd1a0f72aafb17"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.838322 23471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:28.839003 23485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.839180 23471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:28.839272 23471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "601946ce64f74459b92aaaa38b9ebd8d"
format_stamp: "Formatted at 2026-09-24 15:31:28 on dist-test-slave-4g6f"
server_key: "2a249ac2b5d7980e778494d4e46b1fdb"
server_key_iv: "25455bde0075d5b05afd1a0f72aafb17"
server_key_version: "encryptionkey@0"
I20260924 15:31:28.839353 23471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:28.855384 23471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:28.855645 23471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:28.855805 23471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:28.856015 23471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:28.856328 23471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:28.856377 23471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.856410 23471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:28.856436 23471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:28.861689 23471 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:43121
I20260924 15:31:28.861737 23599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:43121 every 8 connection(s)
I20260924 15:31:28.862021 23471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:28.866752 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.862699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:41163 (local address 127.5.98.254:43991)
0924 15:31:28.862759 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.862761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.862985 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:31:28.863408 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.863409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.863420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.863471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.864196 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.864682 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.865577 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.865755 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.865809 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.866495 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.866496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.866496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:28.866500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.866510 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:28.866513 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.866548 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.866624 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.866641 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:28.867004 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.862689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43991 (local address 127.5.98.195:41163)
0924 15:31:28.862880 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.862910 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.863326 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.863520 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.863523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.863532 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:28.864112 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.864117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.864760 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.864760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.865502 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.865504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.865834 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.865835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.865939 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.866381 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:28.866382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:28.866407 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:28.866429 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:28.866562 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:28.866565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:28.866567 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.866616 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.866755 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260924 15:31:28.867235 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23471
I20260924 15:31:28.867388 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:28.867491 23600 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43991
I20260924 15:31:28.867595 23600 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:28.867622 5515 external_mini_cluster.cc:1428] Setting key 000eb0e89ffdb2245daebefece4135f1
I20260924 15:31:28.867852 23600 heartbeater.cc:511] Master 127.5.98.254:43991 requested a full tablet report, sending...
I20260924 15:31:28.868187 23146 ts_manager.cc:194] Registered new tserver with Master: 601946ce64f74459b92aaaa38b9ebd8d (127.5.98.195:43121)
I20260924 15:31:28.868574 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:28.868655 23146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.195:41163
I20260924 15:31:28.872556 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.869945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.5.98.254:43991)
0924 15:31:28.869979 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.869981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.870093 (+ 112us) server_negotiation.cc:407] Connection header received
0924 15:31:28.870143 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.870143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.870150 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.870194 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:28.870313 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.870751 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.871467 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.871635 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.871765 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.872331 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:28.872332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:28.872332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:28.872337 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:28.872350 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:28.872353 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:28.872409 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.872490 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.872508 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:31:28.873946 23146 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:28.875422 23146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:28.881829 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:37169 (local address 127.0.0.1:36180)
0924 15:31:28.877949 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.877957 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.877991 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.878123 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.878125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.878126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.878168 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.878169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879419 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.879420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.880783 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.880786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881536 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.881612 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881626 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.881697 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.881747 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":55,"thread_start_us":137,"threads_started":1}
I20260924 15:31:28.881826 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.5.98.194:37169)
0924 15:31:28.877989 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.877993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.878002 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:31:28.878039 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.878040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.878052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.878117 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.878242 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879336 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.880871 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881455 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.881711 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.881747 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260924 15:31:28.882105 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.5.98.195:43121)
0924 15:31:28.878639 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.878643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.878651 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:31:28.878680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.878682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.878694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.878763 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.878914 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879834 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.880977 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881644 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881805 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.882003 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.882040 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":86,"threads_started":1}
I20260924 15:31:28.882117 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:46119 (local address 127.0.0.1:55032)
0924 15:31:28.877741 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.877781 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.877842 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.878465 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.878466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.878468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.878514 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.878515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879473 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.879474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.880783 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.880786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881833 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.881921 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881927 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.881981 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.882029 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":71,"thread_start_us":45,"threads_started":1}
I20260924 15:31:28.882139 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:43121 (local address 127.0.0.1:54840)
0924 15:31:28.877874 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.877884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.877926 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.878782 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.878783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.878786 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.878830 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.878832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879917 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.879918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.880894 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.880897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881836 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.881921 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881929 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.881985 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.882029 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":9,"thread_start_us":140,"threads_started":1}
I20260924 15:31:28.882464 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.877646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.5.98.193:46119)
0924 15:31:28.878231 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.878235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.878245 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:31:28.878284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.878286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.878308 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.878392 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.878591 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.879357 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.880890 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.881644 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.881715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.881793 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.882340 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.882375 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":126,"threads_started":1}
I20260924 15:31:28.882829 23533 tablet_service.cc:1467] Processing CreateTablet for tablet 6a833f11495b4e9bb64c59c41f67839c (DEFAULT_TABLE table=test-table [id=3a746bae2d2c489ebc69d896631fdd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.883081 23533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a833f11495b4e9bb64c59c41f67839c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.883270 23401 tablet_service.cc:1467] Processing CreateTablet for tablet 6a833f11495b4e9bb64c59c41f67839c (DEFAULT_TABLE table=test-table [id=3a746bae2d2c489ebc69d896631fdd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.883600 23401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a833f11495b4e9bb64c59c41f67839c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.885988 23614 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Bootstrap starting.
I20260924 15:31:28.886319 23615 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Bootstrap starting.
I20260924 15:31:28.886513 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 6a833f11495b4e9bb64c59c41f67839c (DEFAULT_TABLE table=test-table [id=3a746bae2d2c489ebc69d896631fdd7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:28.886804 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a833f11495b4e9bb64c59c41f67839c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:28.886969 23614 tablet_bootstrap.cc:654] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.886965 23615 tablet_bootstrap.cc:654] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.887312 23614 log.cc:824] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.887313 23615 log.cc:824] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.888072 23614 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: No bootstrap required, opened a new log
I20260924 15:31:28.888162 23614 ts_tablet_manager.cc:1397] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:28.888309 23615 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: No bootstrap required, opened a new log
I20260924 15:31:28.888384 23615 ts_tablet_manager.cc:1397] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:28.890108 23618 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: Bootstrap starting.
I20260924 15:31:28.890319 23614 raft_consensus.cc:348] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.890501 23614 raft_consensus.cc:374] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.890540 23614 raft_consensus.cc:729] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601946ce64f74459b92aaaa38b9ebd8d, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.890672 23614 consensus_queue.cc:260] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.890956 23614 ts_tablet_manager.cc:1428] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:28.890869 23615 raft_consensus.cc:348] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.890784 23618 tablet_bootstrap.cc:654] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:28.891024 23615 raft_consensus.cc:374] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.891072 23600 heartbeater.cc:503] Master 127.5.98.254:43991 was elected leader, sending a full tablet report...
I20260924 15:31:28.891067 23615 raft_consensus.cc:729] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d63095084489d93536b25aaf7b1d8, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.891192 23615 consensus_queue.cc:260] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.891319 23618 log.cc:824] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:28.891476 23615 ts_tablet_manager.cc:1428] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:28.891578 23468 heartbeater.cc:503] Master 127.5.98.254:43991 was elected leader, sending a full tablet report...
I20260924 15:31:28.892011 23618 tablet_bootstrap.cc:492] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: No bootstrap required, opened a new log
I20260924 15:31:28.892093 23618 ts_tablet_manager.cc:1397] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:28.894325 23618 raft_consensus.cc:348] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.894482 23618 raft_consensus.cc:374] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:28.894529 23618 raft_consensus.cc:729] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5986dafcf7a6428da337854c48ce5ae1, State: Initialized, Role: FOLLOWER
I20260924 15:31:28.894632 23618 consensus_queue.cc:260] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.894861 23618 ts_tablet_manager.cc:1428] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:28.894965 23336 heartbeater.cc:503] Master 127.5.98.254:43991 was elected leader, sending a full tablet report...
I20260924 15:31:28.912649 23620 raft_consensus.cc:482] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:28.912755 23620 raft_consensus.cc:504] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.913090 23620 leader_election.cc:288] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5986dafcf7a6428da337854c48ce5ae1 (127.5.98.193:46119), 601946ce64f74459b92aaaa38b9ebd8d (127.5.98.195:43121)
I20260924 15:31:28.915772 23612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.913362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:34469 (local address 127.5.98.195:43121)
0924 15:31:28.913415 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.913416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.913430 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:31:28.913449 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.913450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.913457 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.913518 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.913657 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.914098 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915015 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.915412 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915493 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.915501 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.915672 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.915709 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:31:28.915783 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.913230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:43121 (local address 127.5.98.194:34469)
0924 15:31:28.913268 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.913280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.913325 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.913522 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.913523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.913524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.913587 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.913588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.914178 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.914179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.914915 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.914918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.915491 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.915573 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.915576 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.915636 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.915725 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 15:31:28.916016 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.913283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:51689 (local address 127.5.98.193:46119)
0924 15:31:28.913323 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.913325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.913556 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:31:28.913617 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.913618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.913628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.913682 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:28.913817 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.914267 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915220 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.915571 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915695 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.915703 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.915911 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.915951 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:28.916101 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a833f11495b4e9bb64c59c41f67839c" candidate_uuid: "bf1d63095084489d93536b25aaf7b1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601946ce64f74459b92aaaa38b9ebd8d" is_pre_election: true
I20260924 15:31:28.916239 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.913230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:46119 (local address 127.5.98.194:51689)
0924 15:31:28.913484 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.913500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:28.913568 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:28.913685 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:28.913686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:28.913687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:28.913754 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.913756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.914361 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.914362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.915134 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:28.915136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.915696 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.915697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:28.915770 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.915773 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:28.916117 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 15:31:28.916160 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":76,"thread_start_us":134,"threads_started":1}
I20260924 15:31:28.916239 23554 raft_consensus.cc:2454] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d63095084489d93536b25aaf7b1d8 in term 0.
I20260924 15:31:28.916502 23354 leader_election.cc:302] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601946ce64f74459b92aaaa38b9ebd8d, bf1d63095084489d93536b25aaf7b1d8; no voters:
I20260924 15:31:28.916543 23290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a833f11495b4e9bb64c59c41f67839c" candidate_uuid: "bf1d63095084489d93536b25aaf7b1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5986dafcf7a6428da337854c48ce5ae1" is_pre_election: true
I20260924 15:31:28.916633 23620 raft_consensus.cc:2781] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:28.916698 23620 raft_consensus.cc:482] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:28.916697 23290 raft_consensus.cc:2454] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d63095084489d93536b25aaf7b1d8 in term 0.
I20260924 15:31:28.916731 23620 raft_consensus.cc:3037] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.917403 23620 raft_consensus.cc:504] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.917533 23620 leader_election.cc:288] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [CANDIDATE]: Term 1 election: Requested vote from peers 5986dafcf7a6428da337854c48ce5ae1 (127.5.98.193:46119), 601946ce64f74459b92aaaa38b9ebd8d (127.5.98.195:43121)
I20260924 15:31:28.917687 23290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a833f11495b4e9bb64c59c41f67839c" candidate_uuid: "bf1d63095084489d93536b25aaf7b1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5986dafcf7a6428da337854c48ce5ae1"
I20260924 15:31:28.917713 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a833f11495b4e9bb64c59c41f67839c" candidate_uuid: "bf1d63095084489d93536b25aaf7b1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601946ce64f74459b92aaaa38b9ebd8d"
I20260924 15:31:28.917761 23290 raft_consensus.cc:3037] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.917815 23554 raft_consensus.cc:3037] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:28.918443 23290 raft_consensus.cc:2454] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d63095084489d93536b25aaf7b1d8 in term 1.
I20260924 15:31:28.918551 23554 raft_consensus.cc:2454] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d63095084489d93536b25aaf7b1d8 in term 1.
I20260924 15:31:28.918634 23356 leader_election.cc:302] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5986dafcf7a6428da337854c48ce5ae1, bf1d63095084489d93536b25aaf7b1d8; no voters:
I20260924 15:31:28.918764 23620 raft_consensus.cc:2781] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:28.918905 23620 raft_consensus.cc:686] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 1 LEADER]: Becoming Leader. State: Replica: bf1d63095084489d93536b25aaf7b1d8, State: Running, Role: LEADER
I20260924 15:31:28.919014 23620 consensus_queue.cc:237] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } }
I20260924 15:31:28.919797 23146 catalog_manager.cc:5495] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1d63095084489d93536b25aaf7b1d8 (127.5.98.194). New cstate: current_term: 1 leader_uuid: "bf1d63095084489d93536b25aaf7b1d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1d63095084489d93536b25aaf7b1d8" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 } health_report { overall_health: UNKNOWN } } }
I20260924 15:31:28.926723 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.924324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.5.98.194:37169)
0924 15:31:28.924379 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:28.924381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:28.924414 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:31:28.924501 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:28.924502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:28.924519 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:28.924603 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:28.924744 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.925198 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.925942 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:28.926143 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:28.926217 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:28.926393 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:28.926489 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:28.926573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:28.926614 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 15:31:28.926632 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:31:28.928366 23290 raft_consensus.cc:1261] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1 [term 1 FOLLOWER]: Refusing update from remote peer bf1d63095084489d93536b25aaf7b1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:28.928340 23554 raft_consensus.cc:1261] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 1 FOLLOWER]: Refusing update from remote peer bf1d63095084489d93536b25aaf7b1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:28.928755 23624 consensus_queue.cc:1035] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601946ce64f74459b92aaaa38b9ebd8d" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 43121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:28.928892 23620 consensus_queue.cc:1035] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5986dafcf7a6428da337854c48ce5ae1" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:28.931102 23627 mvcc.cc:205] Tried to move back new op lower bound from 7332920889048182784 to 7332920889012944896. Current Snapshot: MvccSnapshot[applied={T|T < 7332920889048182784}]
I20260924 15:31:28.931742 23628 mvcc.cc:205] Tried to move back new op lower bound from 7332920889048182784 to 7332920889012944896. Current Snapshot: MvccSnapshot[applied={T|T < 7332920889048182784}]
I20260924 15:31:28.933709 23146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:28.933808 23146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:28.934521 23146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5941950d7e2a4f7ca6bf7b6aba746f8d: Sending DeleteTablet for 3 replicas of tablet 6a833f11495b4e9bb64c59c41f67839c
I20260924 15:31:28.934813 23269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a833f11495b4e9bb64c59c41f67839c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='slave'} at 127.0.0.1:55032
I20260924 15:31:28.934826 23401 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a833f11495b4e9bb64c59c41f67839c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='slave'} at 127.0.0.1:36180
I20260924 15:31:28.935065 23637 tablet_replica.cc:330] T 6a833f11495b4e9bb64c59c41f67839c P 5986dafcf7a6428da337854c48ce5ae1: stopping tablet replica
I20260924 15:31:28.935060 23533 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a833f11495b4e9bb64c59c41f67839c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:28 UTC) from {username='slave'} at 127.0.0.1:54840
I20260924 15:31:28.935163 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23207
I20260924 15:31:28.935289 23639 tablet_replica.cc:330] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: stopping tablet replica
I20260924 15:31:28.935384 23639 raft_consensus.cc:2229] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:28.935475 23639 raft_consensus.cc:2258] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:28.935894 23638 tablet_replica.cc:330] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: stopping tablet replica
I20260924 15:31:28.936036 23638 raft_consensus.cc:2229] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:31:28.936054 23639 ts_tablet_manager.cc:1905] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:28.936206 23638 raft_consensus.cc:2258] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:28.936625 23638 ts_tablet_manager.cc:1905] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:28.937983 23639 ts_tablet_manager.cc:1918] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:28.938076 23639 log.cc:1196] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/6a833f11495b4e9bb64c59c41f67839c
I20260924 15:31:28.938344 23638 ts_tablet_manager.cc:1918] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:28.938377 23639 ts_tablet_manager.cc:1939] T 6a833f11495b4e9bb64c59c41f67839c P 601946ce64f74459b92aaaa38b9ebd8d: Deleting consensus metadata
I20260924 15:31:28.938495 23638 log.cc:1196] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/6a833f11495b4e9bb64c59c41f67839c
I20260924 15:31:28.938822 23638 ts_tablet_manager.cc:1939] T 6a833f11495b4e9bb64c59c41f67839c P bf1d63095084489d93536b25aaf7b1d8: Deleting consensus metadata
I20260924 15:31:28.939222 23131 catalog_manager.cc:4841] TS bf1d63095084489d93536b25aaf7b1d8 (127.5.98.194:37169): tablet 6a833f11495b4e9bb64c59c41f67839c (table test-table [id=3a746bae2d2c489ebc69d896631fdd7b]) successfully deleted
I20260924 15:31:28.939754 23130 catalog_manager.cc:4841] TS 601946ce64f74459b92aaaa38b9ebd8d (127.5.98.195:43121): tablet 6a833f11495b4e9bb64c59c41f67839c (table test-table [id=3a746bae2d2c489ebc69d896631fdd7b]) successfully deleted
W20260924 15:31:28.942014 23132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:46119 (error 108)
I20260924 15:31:28.942490 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:28.942347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:46119 (local address 127.0.0.1:55038)
0924 15:31:28.942406 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:28.942461 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:31:28.942566 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23339
W20260924 15:31:28.942602 23132 catalog_manager.cc:4584] TS 5986dafcf7a6428da337854c48ce5ae1 (127.5.98.193:46119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a833f11495b4e9bb64c59c41f67839c: Network error: Client connection negotiation failed: client connection to 127.5.98.193:46119: connect: Connection refused (error 111)
I20260924 15:31:28.947455 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23471
I20260924 15:31:28.952262 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23115
2026-09-24T15:31:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (691 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:29 dist-test-slave-4g6f krb5kdc[23643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:29 dist-test-slave-4g6f krb5kdc[23643](info): set up 2 sockets
Sep 24 15:31:29 dist-test-slave-4g6f krb5kdc[23643](info): commencing operation
krb5kdc: starting...
W20260924 15:31:30.004388 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:30.120850 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:37595
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:30.200610 23656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:30.200801 23656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:30.200822 23656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:30.201887 23656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:30.201918 23656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:30.201932 23656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:30.201946 23656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:30.201957 23656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:30.203333 23656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:30.203510 23656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:30.203691 23656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:30.208617 23661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.208701 23664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.208616 23662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:30.208794 23656 server_base.cc:1034] running on GCE node
I20260924 15:31:30.209136 23656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:30.209414 23656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:30.210587 23656 hybrid_clock.cc:648] HybridClock initialized: now 1790263890210566 us; error 54 us; skew 500 ppm
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:30.213106 23656 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:30.213590 23656 webserver.cc:460] Webserver started at http://127.5.98.254:34287/ using document root <none> and password file <none>
I20260924 15:31:30.213828 23656 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:30.213896 23656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:30.214030 23656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:30.215016 23656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "a75bc40fc9a244c4b32950eb05571edb"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "ec4e81d7b9ca6be2562157b99c3b84b8"
server_key_iv: "cb60093aec361fbdd4af5199fb614808"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.215380 23656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "a75bc40fc9a244c4b32950eb05571edb"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "ec4e81d7b9ca6be2562157b99c3b84b8"
server_key_iv: "cb60093aec361fbdd4af5199fb614808"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.216754 23656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:30.217474 23671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.217659 23656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:30.217762 23656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "a75bc40fc9a244c4b32950eb05571edb"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "ec4e81d7b9ca6be2562157b99c3b84b8"
server_key_iv: "cb60093aec361fbdd4af5199fb614808"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.217849 23656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:30.231936 23656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:30.234102 23656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:30.234258 23656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:30.238214 23656 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:37595
I20260924 15:31:30.238250 23724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:37595 every 8 connection(s)
I20260924 15:31:30.238580 23656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:30.239444 23725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:30.241664 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: Bootstrap starting.
I20260924 15:31:30.242249 23725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:30.242542 23725 log.cc:824] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:30.243213 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: No bootstrap required, opened a new log
I20260924 15:31:30.244836 23725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } }
I20260924 15:31:30.244971 23725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:30.244993 23725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75bc40fc9a244c4b32950eb05571edb, State: Initialized, Role: FOLLOWER
I20260924 15:31:30.245186 23725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } }
I20260924 15:31:30.245218 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23656
I20260924 15:31:30.245268 23725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:30.245291 23725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:30.245337 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:30.245357 23725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:30.245523 5515 external_mini_cluster.cc:1428] Setting key c664abfd93e041c87c0b7d93b611ae92
I20260924 15:31:30.245975 23725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } }
I20260924 15:31:30.246088 23725 leader_election.cc:302] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75bc40fc9a244c4b32950eb05571edb; no voters:
I20260924 15:31:30.246249 23725 leader_election.cc:288] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:30.246275 23730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:30.246382 23730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [term 1 LEADER]: Becoming Leader. State: Replica: a75bc40fc9a244c4b32950eb05571edb, State: Running, Role: LEADER
I20260924 15:31:30.246466 23730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } }
I20260924 15:31:30.246546 23725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:30.246923 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75bc40fc9a244c4b32950eb05571edb. Latest consensus state: current_term: 1 leader_uuid: "a75bc40fc9a244c4b32950eb05571edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } } }
I20260924 15:31:30.247190 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:30.247121 23732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75bc40fc9a244c4b32950eb05571edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75bc40fc9a244c4b32950eb05571edb" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 37595 } } }
I20260924 15:31:30.247362 23732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:30.247483 23739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:30.248075 23739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:30.249567 23747 mvcc.cc:205] Tried to move back new op lower bound from 7332920894459904000 to 7332920894449897472. Current Snapshot: MvccSnapshot[applied={T|T < 7332920894459904000}]
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:30.252799 23739 catalog_manager.cc:1347] Generated new cluster ID: 345e952485fd4164b643b49357bad5c8
I20260924 15:31:30.252848 23739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:30.257045 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.245997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.5.98.254:37595)
0924 15:31:30.246166 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.246176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.246212 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:31:30.246342 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.246349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.246408 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.247379 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:30.247513 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.248889 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.249921 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.250369 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.250449 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.253205 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:30.253209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:30.253210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:30.253226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:30.255413 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.255868 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.255871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.255871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.255984 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.256206 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.256207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.256207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.256430 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:30.256601 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.256698 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.256834 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260924 15:31:30.265390 23739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:30.265870 23739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:30.270118 23739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: Generated new TSK 0
I20260924 15:31:30.270294 23739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:31:30.296985 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:30.382387 23751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:30.382589 23751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:30.382612 23751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:30.383780 23751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:30.383818 23751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:30.383844 23751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:30.385267 23751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:30.385506 23751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:30.385759 23751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:30.390739 23759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:30.390766 23751 server_base.cc:1034] running on GCE node
W20260924 15:31:30.390738 23756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.390807 23757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:30.391116 23751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:30.391322 23751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:30.392464 23751 hybrid_clock.cc:648] HybridClock initialized: now 1790263890392452 us; error 30 us; skew 500 ppm
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:30.394977 23751 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:31:30.395344 23751 webserver.cc:460] Webserver started at http://127.5.98.193:39235/ using document root <none> and password file <none>
I20260924 15:31:30.395527 23751 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:30.395567 23751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:30.395650 23751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:30.396699 23751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "782d08f011b24196b6f284b098c49a62"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "996657e53a1106b8524234d2f0ed0ffb"
server_key_iv: "36e2831570fcba2ccb6bb8b1c0400c9c"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.397001 23751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "782d08f011b24196b6f284b098c49a62"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "996657e53a1106b8524234d2f0ed0ffb"
server_key_iv: "36e2831570fcba2ccb6bb8b1c0400c9c"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.398217 23751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:30.398859 23767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.399022 23751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:30.399106 23751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "782d08f011b24196b6f284b098c49a62"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "996657e53a1106b8524234d2f0ed0ffb"
server_key_iv: "36e2831570fcba2ccb6bb8b1c0400c9c"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.399159 23751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:30.449020 23751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:30.451002 23751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:30.451117 23751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:30.451334 23751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:30.451696 23751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:30.451733 23751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.451779 23751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:30.451808 23751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.457485 23751 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:33553
I20260924 15:31:30.457532 23882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:33553 every 8 connection(s)
I20260924 15:31:30.457891 23751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:30.464258 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23751
I20260924 15:31:30.464390 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:30.464603 5515 external_mini_cluster.cc:1428] Setting key b34c7dcf103b2c9278681ef8dac725d1
I20260924 15:31:30.467384 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.458667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:37595 (local address 127.5.98.193:45171)
0924 15:31:30.458835 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.458876 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.459361 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.459598 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.459601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.460232 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:30.460426 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.460432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.461108 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.461109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.461903 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.461905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.462256 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.462256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.462354 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.462858 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:30.462866 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:30.464281 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:30.465958 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.465965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.465967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.466281 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.466478 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.466479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.466479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.466630 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.466922 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:30.466928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:30.467039 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.467121 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.467235 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260924 15:31:30.467381 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.458711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:45171 (local address 127.5.98.254:37595)
0924 15:31:30.458759 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.458761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.458985 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:31:30.459444 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.459445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.459454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.459515 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:30.460514 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.461023 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.461998 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.462185 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.462235 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.464385 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:30.464387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:30.464387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:30.464398 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:30.465873 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.466360 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.466360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.466361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.466413 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.466708 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.466709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.466710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.466853 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:30.466927 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.467168 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.467262 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:31:30.467903 23883 heartbeater.cc:347] Connected to a master server at 127.5.98.254:37595
I20260924 15:31:30.467984 23883 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:30.468118 23883 heartbeater.cc:511] Master 127.5.98.254:37595 requested a full tablet report, sending...
I20260924 15:31:30.468668 23688 ts_manager.cc:194] Registered new tserver with Master: 782d08f011b24196b6f284b098c49a62 (127.5.98.193:33553)
I20260924 15:31:30.469558 23688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:45171
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:31:30.488588 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:30.567384 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:30.567564 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:30.567587 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:30.568783 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:30.568826 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:30.568852 23888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:30.570226 23888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:30.570438 23888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:30.570636 23888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:30.575441 23893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.575479 23894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.575451 23896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:30.575870 23888 server_base.cc:1034] running on GCE node
I20260924 15:31:30.576089 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:30.576313 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:30.577471 23888 hybrid_clock.cc:648] HybridClock initialized: now 1790263890577458 us; error 43 us; skew 500 ppm
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:30.580269 23888 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:31:30.580770 23888 webserver.cc:460] Webserver started at http://127.5.98.194:46087/ using document root <none> and password file <none>
I20260924 15:31:30.581013 23888 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:30.581079 23888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:30.581216 23888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:30.582547 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "b33a5c390cee41f5a200a739e65e37b9"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "381d0cf71426a4d808da189bcb28b9b8"
server_key_iv: "b112c7aa22a8a22ed752ca2a4ffc7333"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.582964 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "b33a5c390cee41f5a200a739e65e37b9"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "381d0cf71426a4d808da189bcb28b9b8"
server_key_iv: "b112c7aa22a8a22ed752ca2a4ffc7333"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.584475 23888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:31:30.585175 23903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.585350 23888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:30.585417 23888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "b33a5c390cee41f5a200a739e65e37b9"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "381d0cf71426a4d808da189bcb28b9b8"
server_key_iv: "b112c7aa22a8a22ed752ca2a4ffc7333"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.585467 23888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:30.593600 23888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:30.595846 23888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:30.596005 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:30.596213 23888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:30.596495 23888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:30.596546 23888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.596585 23888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:30.596611 23888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.602187 23888 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:43027
I20260924 15:31:30.602241 24017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:43027 every 8 connection(s)
I20260924 15:31:30.602571 23888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:30.611292 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.603240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:59549 (local address 127.5.98.254:37595)
0924 15:31:30.603318 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.603321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.603516 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:31:30.604035 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.604036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.604047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.604098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:30.605006 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.605586 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.606739 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.607032 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.607094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.609095 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:30.609096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:30.609096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:30.609103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:30.610215 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.610558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.610558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.610559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.610607 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.610803 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.610803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.610804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.610894 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:30.610941 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.611152 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.611215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:31:30.611490 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.603211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:37595 (local address 127.5.98.194:59549)
0924 15:31:30.603389 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.603427 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.603946 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.604147 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.604149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.604678 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:30.604855 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.604859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.605701 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.605703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.606643 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.606646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.607123 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.607125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.607278 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.607648 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:30.607692 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:30.609007 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:30.610283 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.610289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.610291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.610494 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.610663 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.610663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.610663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.610752 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.610945 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:30.610949 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:30.611052 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.611128 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.611236 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260924 15:31:30.612088 24018 heartbeater.cc:347] Connected to a master server at 127.5.98.254:37595
I20260924 15:31:30.612167 24018 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:30.612295 24018 heartbeater.cc:511] Master 127.5.98.254:37595 requested a full tablet report, sending...
I20260924 15:31:30.612676 23688 ts_manager.cc:194] Registered new tserver with Master: b33a5c390cee41f5a200a739e65e37b9 (127.5.98.194:43027)
I20260924 15:31:30.612809 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 23888
I20260924 15:31:30.612932 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:30.613122 5515 external_mini_cluster.cc:1428] Setting key 123726dd3e0c8ef222f032b1e1029392
I20260924 15:31:30.613126 23688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:59549
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:31:30.636637 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:30.716625 24023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:30.716799 24023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:30.716821 24023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:30.717890 24023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:30.717923 24023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:30.717948 24023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:30.719228 24023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:30.719390 24023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:30.719553 24023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:30.724452 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.724476 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:30.724781 24031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:30.724787 24023 server_base.cc:1034] running on GCE node
I20260924 15:31:30.725023 24023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:30.725227 24023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:30.726361 24023 hybrid_clock.cc:648] HybridClock initialized: now 1790263890726346 us; error 31 us; skew 500 ppm
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:30.728864 24023 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:31:30.729318 24023 webserver.cc:460] Webserver started at http://127.5.98.195:36477/ using document root <none> and password file <none>
I20260924 15:31:30.729526 24023 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:30.729568 24023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:30.729707 24023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:30.730671 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "680260aae08549d08a4c65fa140aa8c0"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "38deb705cd2b0aebaeedac044945b7d0"
server_key_iv: "29793fac11f4186e29c855a6d4d58e74"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.731068 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "680260aae08549d08a4c65fa140aa8c0"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "38deb705cd2b0aebaeedac044945b7d0"
server_key_iv: "29793fac11f4186e29c855a6d4d58e74"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.732568 24023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:30.733331 24038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.733505 24023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:30.733604 24023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "680260aae08549d08a4c65fa140aa8c0"
format_stamp: "Formatted at 2026-09-24 15:31:30 on dist-test-slave-4g6f"
server_key: "38deb705cd2b0aebaeedac044945b7d0"
server_key_iv: "29793fac11f4186e29c855a6d4d58e74"
server_key_version: "encryptionkey@0"
I20260924 15:31:30.733683 24023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:30.748068 24023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:30.750052 24023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:30.750213 24023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:30.750466 24023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:30.750773 24023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:30.750805 24023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.750828 24023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:30.750874 24023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:30.756524 24023 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:44009
I20260924 15:31:30.756613 24152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:44009 every 8 connection(s)
I20260924 15:31:30.756907 24023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:30.760883 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24023
I20260924 15:31:30.761023 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:30.761211 5515 external_mini_cluster.cc:1428] Setting key 12f49d2fe70120c184c7862e636f9dfa
Sep 24 15:31:30 dist-test-slave-4g6f krb5kdc[23643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263890, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:30.765980 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.757704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:53077 (local address 127.5.98.254:37595)
0924 15:31:30.757766 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.757768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.757926 (+ 158us) server_negotiation.cc:407] Connection header received
0924 15:31:30.758425 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.758427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.758439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.758508 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:30.759410 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.759990 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.761018 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.761298 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.761372 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.763626 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:30.763628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:30.763628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:30.763635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:30.764860 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.765225 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.765226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.765226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.765275 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.765450 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.765450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.765450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.765545 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:30.765594 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.765837 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.765898 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:31:30.766129 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.757629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:37595 (local address 127.5.98.195:53077)
0924 15:31:30.757801 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.757841 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.758307 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.758598 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.758602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.759131 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:30.759310 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.759314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.760085 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.760086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.760909 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.760911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.761383 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.761384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.761539 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.762047 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:30.762055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:30.763538 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:30.764933 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.764940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.764941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.765159 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.765319 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:30.765319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:30.765320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:30.765401 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:30.765602 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:30.765607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:30.765724 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.765803 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.765913 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260924 15:31:30.766613 24153 heartbeater.cc:347] Connected to a master server at 127.5.98.254:37595
I20260924 15:31:30.766743 24153 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:30.766939 24153 heartbeater.cc:511] Master 127.5.98.254:37595 requested a full tablet report, sending...
I20260924 15:31:30.767331 23688 ts_manager.cc:194] Registered new tserver with Master: 680260aae08549d08a4c65fa140aa8c0 (127.5.98.195:44009)
I20260924 15:31:30.767783 23688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:53077
I20260924 15:31:30.772790 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:30.780803 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.774088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.5.98.254:37595)
0924 15:31:30.774128 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.774130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.774229 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:31:30.774280 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.774281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.774288 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.774335 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:30.774850 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.775407 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.776458 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.776756 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.776827 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.778245 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:30.778246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:30.778247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:30.778254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:30.779560 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.779984 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.779985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.779986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.780038 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:30.780274 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:30.780275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:30.780275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:30.780415 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:30.780525 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.780652 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.780713 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:31:30.782179 23688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:30.783658 23688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:30.790107 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.785927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:43027 (local address 127.0.0.1:39646)
0924 15:31:30.786313 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.786322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.786380 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.786660 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.786661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.786663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:30.786698 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.786700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.788004 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.788006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.788996 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.788998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.789822 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.789823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.789894 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.789907 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.789980 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.790031 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":85,"thread_start_us":90,"threads_started":1}
I20260924 15:31:30.790141 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.786164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.5.98.194:43027)
0924 15:31:30.786486 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.786489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.786497 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:31:30.786555 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.786557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.786571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.786649 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:30.786759 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.787881 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.789096 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.789732 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.789805 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.789855 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.790012 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.790052 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":70,"threads_started":1}
I20260924 15:31:30.790452 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.785909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33553 (local address 127.0.0.1:48166)
0924 15:31:30.786070 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.786109 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.786174 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.786551 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.786554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.786556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:30.786637 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.786639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.788027 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.788028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.789291 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.789293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.790200 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.790202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.790277 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.790281 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.790311 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.790358 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":42,"thread_start_us":57,"threads_started":1}
I20260924 15:31:30.790639 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.785909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:44009 (local address 127.0.0.1:38294)
0924 15:31:30.786277 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.786285 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.786356 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.786558 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.786559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.786561 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:30.786802 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.786803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.787941 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.787942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.789273 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.789276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.790197 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.790198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.790263 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.790267 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.790297 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.790355 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":214,"spinlock_wait_cycles":384,"thread_start_us":108,"threads_started":1}
I20260924 15:31:30.790642 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.786120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.5.98.195:44009)
0924 15:31:30.786278 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.786282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.786349 (+ 67us) server_negotiation.cc:407] Connection header received
0924 15:31:30.786426 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.786427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.786444 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.786956 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:30.787019 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.787824 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.789404 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.790085 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.790420 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.790492 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.790517 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.790551 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260924 15:31:30.791038 23951 tablet_service.cc:1467] Processing CreateTablet for tablet ee77b57e85eb4a45a417696ab130397b (DEFAULT_TABLE table=test-table [id=a933e14b451545b0a17db7ef663a1c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:30.791186 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.786173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.5.98.193:33553)
0924 15:31:30.786380 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.786384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.786394 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:31:30.786431 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.786433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.786447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.786534 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:30.786726 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.787939 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.789386 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.790082 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.790977 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.791045 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.791072 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.791113 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260924 15:31:30.791236 24086 tablet_service.cc:1467] Processing CreateTablet for tablet ee77b57e85eb4a45a417696ab130397b (DEFAULT_TABLE table=test-table [id=a933e14b451545b0a17db7ef663a1c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:30.791419 23951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee77b57e85eb4a45a417696ab130397b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:30.791566 24086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee77b57e85eb4a45a417696ab130397b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:30.791781 23816 tablet_service.cc:1467] Processing CreateTablet for tablet ee77b57e85eb4a45a417696ab130397b (DEFAULT_TABLE table=test-table [id=a933e14b451545b0a17db7ef663a1c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:30.792085 23816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee77b57e85eb4a45a417696ab130397b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:30.794099 24170 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Bootstrap starting.
I20260924 15:31:30.794847 24170 tablet_bootstrap.cc:654] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:30.795017 24171 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Bootstrap starting.
I20260924 15:31:30.795342 24172 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Bootstrap starting.
I20260924 15:31:30.795475 24170 log.cc:824] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:30.795728 24171 tablet_bootstrap.cc:654] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:30.796061 24172 tablet_bootstrap.cc:654] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:30.796298 24171 log.cc:824] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:30.796360 24170 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: No bootstrap required, opened a new log
I20260924 15:31:30.796415 24170 ts_tablet_manager.cc:1397] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:31:30.796460 24172 log.cc:824] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:30.797043 24171 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: No bootstrap required, opened a new log
I20260924 15:31:30.797101 24171 ts_tablet_manager.cc:1397] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:31:30.797226 24172 tablet_bootstrap.cc:492] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: No bootstrap required, opened a new log
I20260924 15:31:30.797276 24172 ts_tablet_manager.cc:1397] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:30.797945 24170 raft_consensus.cc:348] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.798126 24170 raft_consensus.cc:374] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:30.798171 24170 raft_consensus.cc:729] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680260aae08549d08a4c65fa140aa8c0, State: Initialized, Role: FOLLOWER
I20260924 15:31:30.798286 24170 consensus_queue.cc:260] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.798580 24170 ts_tablet_manager.cc:1428] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:31:30.798584 24153 heartbeater.cc:503] Master 127.5.98.254:37595 was elected leader, sending a full tablet report...
I20260924 15:31:30.799113 24172 raft_consensus.cc:348] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.799089 24171 raft_consensus.cc:348] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.799271 24171 raft_consensus.cc:374] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:30.799271 24172 raft_consensus.cc:374] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:30.799348 24171 raft_consensus.cc:729] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33a5c390cee41f5a200a739e65e37b9, State: Initialized, Role: FOLLOWER
I20260924 15:31:30.799348 24172 raft_consensus.cc:729] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782d08f011b24196b6f284b098c49a62, State: Initialized, Role: FOLLOWER
I20260924 15:31:30.799456 24171 consensus_queue.cc:260] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.799477 24172 consensus_queue.cc:260] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.799759 24171 ts_tablet_manager.cc:1428] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:31:30.799759 24172 ts_tablet_manager.cc:1428] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:31:30.799913 24018 heartbeater.cc:503] Master 127.5.98.254:37595 was elected leader, sending a full tablet report...
I20260924 15:31:30.799913 23883 heartbeater.cc:503] Master 127.5.98.254:37595 was elected leader, sending a full tablet report...
W20260924 15:31:30.853135 24019 tablet.cc:2369] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:30.930879 24176 raft_consensus.cc:482] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:30.931005 24176 raft_consensus.cc:504] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.931447 24176 leader_election.cc:288] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 782d08f011b24196b6f284b098c49a62 (127.5.98.193:33553), b33a5c390cee41f5a200a739e65e37b9 (127.5.98.194:43027)
I20260924 15:31:30.934227 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.931796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:51293 (local address 127.5.98.193:33553)
0924 15:31:30.931839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.931841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.931848 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:31:30.931881 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.931881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.931890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.931959 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:30.932120 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.932594 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.933500 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.933866 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.933955 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.933964 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.934124 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.934163 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:31:30.934227 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.931629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33553 (local address 127.5.98.195:51293)
0924 15:31:30.931700 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.931715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.931770 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.931964 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.931965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.931967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:30.932042 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.932044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.932676 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.932677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.933401 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.933403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.933956 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.933957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.934035 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.934038 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.934114 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.934157 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 15:31:30.934417 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.931629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:43027 (local address 127.5.98.195:58389)
0924 15:31:30.931928 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.931936 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:30.932014 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:30.932166 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:30.932167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:30.932169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:30.932245 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.932246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.932852 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.932853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.933587 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:30.933589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.934148 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.934149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:30.934226 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.934230 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:30.934298 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:31:30.934336 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":93,"thread_start_us":129,"threads_started":1}
I20260924 15:31:30.934403 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.931766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:58389 (local address 127.5.98.194:43027)
0924 15:31:30.931817 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.931819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.932004 (+ 185us) server_negotiation.cc:407] Connection header received
0924 15:31:30.932072 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.932073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.932087 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.932162 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:30.932317 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.932766 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.933682 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.934060 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.934146 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.934154 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.934304 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.934340 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:31:30.934592 23836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee77b57e85eb4a45a417696ab130397b" candidate_uuid: "680260aae08549d08a4c65fa140aa8c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782d08f011b24196b6f284b098c49a62" is_pre_election: true
I20260924 15:31:30.934791 23836 raft_consensus.cc:2454] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680260aae08549d08a4c65fa140aa8c0 in term 0.
I20260924 15:31:30.934756 23971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee77b57e85eb4a45a417696ab130397b" candidate_uuid: "680260aae08549d08a4c65fa140aa8c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33a5c390cee41f5a200a739e65e37b9" is_pre_election: true
I20260924 15:31:30.934883 23971 raft_consensus.cc:2454] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680260aae08549d08a4c65fa140aa8c0 in term 0.
I20260924 15:31:30.935029 24040 leader_election.cc:302] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680260aae08549d08a4c65fa140aa8c0, 782d08f011b24196b6f284b098c49a62; no voters:
I20260924 15:31:30.935221 24176 raft_consensus.cc:2781] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:30.935276 24176 raft_consensus.cc:482] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:30.935312 24176 raft_consensus.cc:3037] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:30.936092 24176 raft_consensus.cc:504] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.936333 24176 leader_election.cc:288] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [CANDIDATE]: Term 1 election: Requested vote from peers 782d08f011b24196b6f284b098c49a62 (127.5.98.193:33553), b33a5c390cee41f5a200a739e65e37b9 (127.5.98.194:43027)
I20260924 15:31:30.936390 23836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee77b57e85eb4a45a417696ab130397b" candidate_uuid: "680260aae08549d08a4c65fa140aa8c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782d08f011b24196b6f284b098c49a62"
I20260924 15:31:30.936492 23836 raft_consensus.cc:3037] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:30.936530 23971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee77b57e85eb4a45a417696ab130397b" candidate_uuid: "680260aae08549d08a4c65fa140aa8c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33a5c390cee41f5a200a739e65e37b9"
I20260924 15:31:30.936609 23971 raft_consensus.cc:3037] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:30.937218 23836 raft_consensus.cc:2454] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680260aae08549d08a4c65fa140aa8c0 in term 1.
I20260924 15:31:30.937345 23971 raft_consensus.cc:2454] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680260aae08549d08a4c65fa140aa8c0 in term 1.
I20260924 15:31:30.937444 24040 leader_election.cc:302] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680260aae08549d08a4c65fa140aa8c0, 782d08f011b24196b6f284b098c49a62; no voters:
I20260924 15:31:30.937582 24176 raft_consensus.cc:2781] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:30.937781 24176 raft_consensus.cc:686] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 1 LEADER]: Becoming Leader. State: Replica: 680260aae08549d08a4c65fa140aa8c0, State: Running, Role: LEADER
I20260924 15:31:30.937877 24176 consensus_queue.cc:237] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } }
I20260924 15:31:30.938727 23687 catalog_manager.cc:5495] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 680260aae08549d08a4c65fa140aa8c0 (127.5.98.195). New cstate: current_term: 1 leader_uuid: "680260aae08549d08a4c65fa140aa8c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "680260aae08549d08a4c65fa140aa8c0" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 44009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 } health_report { overall_health: UNKNOWN } } }
W20260924 15:31:30.958767 23884 tablet.cc:2369] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:30.970845 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.968563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.5.98.195:44009)
0924 15:31:30.968610 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:30.968612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:30.968619 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:31:30.968655 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:30.968658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:30.968679 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:30.968752 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:30.968880 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.969321 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.970081 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:30.970290 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:30.970378 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:30.970512 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:30.970604 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:30.970708 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:30.970761 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 15:31:30.970769 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:31:30.972636 23971 raft_consensus.cc:1261] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 1 FOLLOWER]: Refusing update from remote peer 680260aae08549d08a4c65fa140aa8c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:30.972648 23836 raft_consensus.cc:1261] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 1 FOLLOWER]: Refusing update from remote peer 680260aae08549d08a4c65fa140aa8c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:30.972995 24180 consensus_queue.cc:1035] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33a5c390cee41f5a200a739e65e37b9" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:30.973158 24176 consensus_queue.cc:1035] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782d08f011b24196b6f284b098c49a62" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:30.975459 24183 mvcc.cc:205] Tried to move back new op lower bound from 7332920897421332480 to 7332920897282379776. Current Snapshot: MvccSnapshot[applied={T|T < 7332920897421332480}]
I20260924 15:31:30.975798 24184 mvcc.cc:205] Tried to move back new op lower bound from 7332920897421332480 to 7332920897282379776. Current Snapshot: MvccSnapshot[applied={T|T < 7332920897421332480}]
I20260924 15:31:30.975863 24185 mvcc.cc:205] Tried to move back new op lower bound from 7332920897421332480 to 7332920897282379776. Current Snapshot: MvccSnapshot[applied={T|T < 7332920897421332480}]
I20260924 15:31:30.977736 23688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:30.977836 23688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:30.978511 23688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a75bc40fc9a244c4b32950eb05571edb: Sending DeleteTablet for 3 replicas of tablet ee77b57e85eb4a45a417696ab130397b
I20260924 15:31:30.978829 23816 tablet_service.cc:1514] Processing DeleteTablet for tablet ee77b57e85eb4a45a417696ab130397b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:30 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:48166
I20260924 15:31:30.978832 24086 tablet_service.cc:1514] Processing DeleteTablet for tablet ee77b57e85eb4a45a417696ab130397b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:30 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:38294
I20260924 15:31:30.979074 24193 tablet_replica.cc:330] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: stopping tablet replica
I20260924 15:31:30.979058 23951 tablet_service.cc:1514] Processing DeleteTablet for tablet ee77b57e85eb4a45a417696ab130397b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:30 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:39646
I20260924 15:31:30.979182 24193 raft_consensus.cc:2229] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:30.979308 24193 raft_consensus.cc:2258] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:30.979312 24195 tablet_replica.cc:330] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: stopping tablet replica
I20260924 15:31:30.979506 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23751
I20260924 15:31:30.979516 24195 raft_consensus.cc:2229] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:30.979696 24195 raft_consensus.cc:2258] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:30.979771 24193 ts_tablet_manager.cc:1905] T ee77b57e85eb4a45a417696ab130397b P 782d08f011b24196b6f284b098c49a62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:30.981221 24195 ts_tablet_manager.cc:1905] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:30.982983 24195 ts_tablet_manager.cc:1918] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:30.983124 24195 log.cc:1196] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/ee77b57e85eb4a45a417696ab130397b
I20260924 15:31:30.983466 24195 ts_tablet_manager.cc:1939] T ee77b57e85eb4a45a417696ab130397b P b33a5c390cee41f5a200a739e65e37b9: Deleting consensus metadata
I20260924 15:31:30.983945 23675 catalog_manager.cc:4841] TS b33a5c390cee41f5a200a739e65e37b9 (127.5.98.194:43027): tablet ee77b57e85eb4a45a417696ab130397b (table test-table [id=a933e14b451545b0a17db7ef663a1c69]) successfully deleted
W20260924 15:31:30.986370 23673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:33553 (error 108)
I20260924 15:31:30.986443 24194 tablet_replica.cc:330] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: stopping tablet replica
I20260924 15:31:30.986538 24194 raft_consensus.cc:2229] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:31:30.986667 24194 raft_consensus.cc:2258] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:30.986918 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:30.986771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33553 (local address 127.0.0.1:48178)
0924 15:31:30.986824 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:30.986874 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:31:30.987033 23673 catalog_manager.cc:4584] TS 782d08f011b24196b6f284b098c49a62 (127.5.98.193:33553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee77b57e85eb4a45a417696ab130397b: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33553: connect: Connection refused (error 111)
I20260924 15:31:30.987111 24194 ts_tablet_manager.cc:1905] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:30.987387 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23888
I20260924 15:31:30.989094 24194 ts_tablet_manager.cc:1918] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:30.989171 24194 log.cc:1196] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/ee77b57e85eb4a45a417696ab130397b
I20260924 15:31:30.989463 24194 ts_tablet_manager.cc:1939] T ee77b57e85eb4a45a417696ab130397b P 680260aae08549d08a4c65fa140aa8c0: Deleting consensus metadata
I20260924 15:31:30.989862 23672 catalog_manager.cc:4841] TS 680260aae08549d08a4c65fa140aa8c0 (127.5.98.195:44009): tablet ee77b57e85eb4a45a417696ab130397b (table test-table [id=a933e14b451545b0a17db7ef663a1c69]) successfully deleted
I20260924 15:31:30.993463 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24023
I20260924 15:31:30.998142 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 23656
2026-09-24T15:31:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2046 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T15:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:31Z Disabled control of system clock
I20260924 15:31:31.026036 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34481
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:31.108433 24200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:31.108621 24200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:31.108641 24200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:31.109791 24200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:31.109833 24200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:31.109845 24200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:31.109876 24200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:31.109901 24200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:31.111181 24200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:31.111440 24200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:31.111742 24200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:31.116624 24208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:31.116613 24205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:31.116613 24206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.116969 24200 server_base.cc:1034] running on GCE node
I20260924 15:31:31.117220 24200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:31.117502 24200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:31.118643 24200 hybrid_clock.cc:648] HybridClock initialized: now 1790263891118627 us; error 33 us; skew 500 ppm
I20260924 15:31:31.120050 24200 webserver.cc:460] Webserver started at http://127.5.98.254:45255/ using document root <none> and password file <none>
I20260924 15:31:31.120319 24200 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:31.120368 24200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:31.120519 24200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:31.121589 24200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "8ac6ac51f91743ec8004c90fa58bcd65"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "037ee69c79355622051a9d9cc423baac"
server_key_iv: "40016d6f8515f39e36412fc8518f49a0"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.121958 24200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "8ac6ac51f91743ec8004c90fa58bcd65"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "037ee69c79355622051a9d9cc423baac"
server_key_iv: "40016d6f8515f39e36412fc8518f49a0"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.123224 24200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:31.124012 24214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.124199 24200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.124289 24200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "8ac6ac51f91743ec8004c90fa58bcd65"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "037ee69c79355622051a9d9cc423baac"
server_key_iv: "40016d6f8515f39e36412fc8518f49a0"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.124365 24200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:31.135356 24200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:31.135629 24200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:31.135826 24200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:31.139631 24200 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34481
I20260924 15:31:31.139700 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34481 every 8 connection(s)
I20260924 15:31:31.140094 24200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:31.141022 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:31.143162 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: Bootstrap starting.
I20260924 15:31:31.143790 24267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:31.144079 24267 log.cc:824] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:31.144724 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: No bootstrap required, opened a new log
I20260924 15:31:31.146258 24267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } }
I20260924 15:31:31.146442 24267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:31.146516 24267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6ac51f91743ec8004c90fa58bcd65, State: Initialized, Role: FOLLOWER
I20260924 15:31:31.146663 24267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } }
I20260924 15:31:31.146746 24267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:31.146785 24267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:31.146836 24267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:31.147322 24267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } }
I20260924 15:31:31.147454 24267 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac6ac51f91743ec8004c90fa58bcd65; no voters:
I20260924 15:31:31.147629 24267 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:31.147706 24270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:31.147883 24270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6ac51f91743ec8004c90fa58bcd65, State: Running, Role: LEADER
I20260924 15:31:31.147943 24267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:31.148007 24270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } }
I20260924 15:31:31.148417 24272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac6ac51f91743ec8004c90fa58bcd65. Latest consensus state: current_term: 1 leader_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } } }
I20260924 15:31:31.148521 24272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:31.148500 24271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac6ac51f91743ec8004c90fa58bcd65" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34481 } } }
I20260924 15:31:31.148761 24271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:31.149003 24279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:31.149535 24279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:31.150190 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24200
I20260924 15:31:31.150300 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:31.150502 5515 external_mini_cluster.cc:1428] Setting key 2954ccb6531f7c082f30b7b6ee099086
I20260924 15:31:31.151234 24279 catalog_manager.cc:1347] Generated new cluster ID: e6197148cd68431d8460f92d87276d7e
I20260924 15:31:31.151288 24279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:31.156606 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.151143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49808 (local address 127.5.98.254:34481)
0924 15:31:31.151325 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.151333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.151363 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:31:31.151483 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.151489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.151551 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.151696 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:31.151967 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.153279 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.154436 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.155057 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.155175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.155921 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:31.155924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:31.155925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:31.156129 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:31.156167 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:31.156180 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:31.156241 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.156339 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.156416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 15:31:31.162879 24279 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:31.163352 24279 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:31.167801 24279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: Generated new TSK 0
I20260924 15:31:31.167977 24279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:31.171779 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--builtin_ntp_servers=127.5.98.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:31.258912 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:31.259099 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:31.259120 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:31.260258 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:31.260296 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:31.260320 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:31.261651 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:31.261885 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:31.262192 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:31.267937 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.268077 24291 server_base.cc:1034] running on GCE node
W20260924 15:31:31.267925 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:31.268201 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.268443 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:31.268678 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:31.269843 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790263891269800 us; error 60 us; skew 500 ppm
I20260924 15:31:31.271113 24291 webserver.cc:460] Webserver started at http://127.5.98.193:33853/ using document root <none> and password file <none>
I20260924 15:31:31.271358 24291 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:31.271425 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:31.271575 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:31.272691 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "7bb566bc5bb94d7187a41cb837844500"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d801de61f522ccab451ae91a499f5421"
server_key_iv: "da912dce5b6a3878eb14b6a060694b6b"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.273041 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "7bb566bc5bb94d7187a41cb837844500"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d801de61f522ccab451ae91a499f5421"
server_key_iv: "da912dce5b6a3878eb14b6a060694b6b"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.274340 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:31.275143 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.275362 24291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:31.275486 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "7bb566bc5bb94d7187a41cb837844500"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d801de61f522ccab451ae91a499f5421"
server_key_iv: "da912dce5b6a3878eb14b6a060694b6b"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.275561 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:31.286571 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:31.286840 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:31.286996 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:31.287205 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:31.287508 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:31.287563 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.287602 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:31.287631 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.293186 24291 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:33085
I20260924 15:31:31.293222 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:33085 every 8 connection(s)
I20260924 15:31:31.293560 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:31:31.297065 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24291
I20260924 15:31:31.297195 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:31.297436 5515 external_mini_cluster.cc:1428] Setting key f22bf44bdf08e6816f30c33063b57e0b
I20260924 15:31:31.298573 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--builtin_ntp_servers=127.5.98.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:31:31.298776 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.294239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:57487 (local address 127.5.98.254:34481)
0924 15:31:31.294283 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.294285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.294518 (+ 233us) server_negotiation.cc:407] Connection header received
0924 15:31:31.294959 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.294960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.294971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.295021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:31.295743 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.296252 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.297226 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.297490 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.297570 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.298475 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:31.298477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:31.298477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:31.298482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:31.298496 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:31.298501 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:31.298550 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.298648 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.298699 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:31:31.298993 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.294237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34481 (local address 127.5.98.193:57487)
0924 15:31:31.294401 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.294432 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.294881 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.295099 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.295102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.295110 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:31.295576 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.295580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.296346 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.296347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.297121 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.297123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.297578 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.297579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.297710 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.298331 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:31.298331 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:31.298359 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:31.298379 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:31.298559 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:31.298565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:31.298568 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.298647 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.298764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 15:31:31.299408 24424 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34481
I20260924 15:31:31.299511 24424 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:31.299747 24424 heartbeater.cc:511] Master 127.5.98.254:34481 requested a full tablet report, sending...
I20260924 15:31:31.300246 24231 ts_manager.cc:194] Registered new tserver with Master: 7bb566bc5bb94d7187a41cb837844500 (127.5.98.193:33085)
I20260924 15:31:31.300860 24231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.193:57487
W20260924 15:31:31.377488 24427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:31.377663 24427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:31.377683 24427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:31.378836 24427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:31.378880 24427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:31.378906 24427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:31.380365 24427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:31.380550 24427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:31.380779 24427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:31.385638 24432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.385735 24427 server_base.cc:1034] running on GCE node
W20260924 15:31:31.385675 24435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:31.385890 24433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.386124 24427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:31.386399 24427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:31.387576 24427 hybrid_clock.cc:648] HybridClock initialized: now 1790263891387565 us; error 29 us; skew 500 ppm
I20260924 15:31:31.388842 24427 webserver.cc:460] Webserver started at http://127.5.98.194:39463/ using document root <none> and password file <none>
I20260924 15:31:31.389091 24427 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:31.389160 24427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:31.389312 24427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:31.390362 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "d44c879a4613472a9bdba59ea5c35fab"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d552a8042d974cd39813c72e2a5ecd4e"
server_key_iv: "ac7982e55f4332c82b94d067572073e9"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.390726 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "d44c879a4613472a9bdba59ea5c35fab"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d552a8042d974cd39813c72e2a5ecd4e"
server_key_iv: "ac7982e55f4332c82b94d067572073e9"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.392041 24427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:31.392712 24441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.392903 24427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:31.393011 24427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "d44c879a4613472a9bdba59ea5c35fab"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "d552a8042d974cd39813c72e2a5ecd4e"
server_key_iv: "ac7982e55f4332c82b94d067572073e9"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.393092 24427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:31.410874 24427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:31.411145 24427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:31.411316 24427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:31.411530 24427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:31.411849 24427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:31.411913 24427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.411953 24427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:31.411979 24427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.417722 24427 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:44897
I20260924 15:31:31.417805 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:44897 every 8 connection(s)
I20260924 15:31:31.418111 24427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:31:31.423029 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.418840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:48225 (local address 127.5.98.254:34481)
0924 15:31:31.418904 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.418906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.419108 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:31:31.419548 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.419549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.419558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.419615 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:31.420338 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.420835 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.421740 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.421920 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.421970 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.422804 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:31.422804 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:31.422805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:31.422809 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:31.422819 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:31.422822 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:31.422859 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.422951 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.422974 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:31:31.423203 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.418798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34481 (local address 127.5.98.194:48225)
0924 15:31:31.418977 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.419015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.419463 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.419763 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.419772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.419781 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:31.420260 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.420264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.420915 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.420916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.421665 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.421667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.421995 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.421996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.422105 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.422663 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:31.422664 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:31.422691 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:31.422713 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:31.422877 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:31.422882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:31.422884 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.422937 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.423013 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260924 15:31:31.423645 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24427
I20260924 15:31:31.423727 24556 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34481
I20260924 15:31:31.423771 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:31.423821 24556 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:31.423983 5515 external_mini_cluster.cc:1428] Setting key ff78822e07bd66f9b239ed040074e764
I20260924 15:31:31.423988 24556 heartbeater.cc:511] Master 127.5.98.254:34481 requested a full tablet report, sending...
I20260924 15:31:31.424407 24230 ts_manager.cc:194] Registered new tserver with Master: d44c879a4613472a9bdba59ea5c35fab (127.5.98.194:44897)
I20260924 15:31:31.424901 24230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.194:48225
I20260924 15:31:31.424928 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--builtin_ntp_servers=127.5.98.212:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:31.504953 24559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:31.505139 24559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:31.505159 24559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:31.506328 24559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:31.506367 24559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:31.506390 24559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:31.507771 24559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:31.508030 24559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:31.508330 24559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:31.513839 24564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:31.513833 24567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.513998 24559 server_base.cc:1034] running on GCE node
W20260924 15:31:31.513832 24565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:31.514264 24559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:31.514518 24559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:31.515717 24559 hybrid_clock.cc:648] HybridClock initialized: now 1790263891515703 us; error 49 us; skew 500 ppm
I20260924 15:31:31.516870 24559 webserver.cc:460] Webserver started at http://127.5.98.195:42255/ using document root <none> and password file <none>
I20260924 15:31:31.517104 24559 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:31.517171 24559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:31.517300 24559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:31.518354 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "4c75fd9c755f491c82966e3b72d663c2"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "64608a78b93e743e262605cf2b46a62b"
server_key_iv: "8aa3732538f0ad4d8cc297c72232443f"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.518719 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "4c75fd9c755f491c82966e3b72d663c2"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "64608a78b93e743e262605cf2b46a62b"
server_key_iv: "8aa3732538f0ad4d8cc297c72232443f"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.520053 24559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:31.520862 24573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.521029 24559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:31.521130 24559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "4c75fd9c755f491c82966e3b72d663c2"
format_stamp: "Formatted at 2026-09-24 15:31:31 on dist-test-slave-4g6f"
server_key: "64608a78b93e743e262605cf2b46a62b"
server_key_iv: "8aa3732538f0ad4d8cc297c72232443f"
server_key_version: "encryptionkey@0"
I20260924 15:31:31.521202 24559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:31.546708 24559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:31.546998 24559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:31.547158 24559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:31.547385 24559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:31.547773 24559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:31.547817 24559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.547857 24559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:31.547883 24559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:31.553699 24559 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:46469
I20260924 15:31:31.553746 24687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:46469 every 8 connection(s)
I20260924 15:31:31.554039 24559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:31.558804 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.554686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:59133 (local address 127.5.98.254:34481)
0924 15:31:31.554776 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.554779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.554942 (+ 163us) server_negotiation.cc:407] Connection header received
0924 15:31:31.555434 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.555436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.555446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.555500 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:31.556136 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.556640 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.557579 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.557770 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.557822 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.558572 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:31.558574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:31.558574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:31.558579 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:31.558593 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:31.558597 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:31.558653 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.558721 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.558745 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:31:31.558956 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.554644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:34481 (local address 127.5.98.195:59133)
0924 15:31:31.554826 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.554858 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.555334 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.555550 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.555554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.555561 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:31.556052 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.556056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.556732 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.556733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.557501 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.557503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.557842 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.557843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.557948 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.558398 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:31.558399 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:31.558426 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:31.558475 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:31.558655 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:31.558657 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:31.558659 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.558704 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.558777 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260924 15:31:31.559466 24688 heartbeater.cc:347] Connected to a master server at 127.5.98.254:34481
I20260924 15:31:31.559569 24688 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:31.559760 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24559
I20260924 15:31:31.559830 24688 heartbeater.cc:511] Master 127.5.98.254:34481 requested a full tablet report, sending...
I20260924 15:31:31.559849 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:31.560060 5515 external_mini_cluster.cc:1428] Setting key 4e4aa05293145e140c0c2fe5016c8c01
I20260924 15:31:31.560287 24230 ts_manager.cc:194] Registered new tserver with Master: 4c75fd9c755f491c82966e3b72d663c2 (127.5.98.195:46469)
I20260924 15:31:31.560724 24230 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.195:59133
I20260924 15:31:31.560927 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:31.565151 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.562219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.5.98.254:34481)
0924 15:31:31.562278 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.562280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.562408 (+ 128us) server_negotiation.cc:407] Connection header received
0924 15:31:31.562474 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.562475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.562486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.562553 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:31.562732 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.563190 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.563944 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.564126 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.564205 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.564870 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:31.564871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:31.564871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:31.564875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:31.564887 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:31.564891 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:31.564997 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.565071 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.565082 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:31:31.566812 24230 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:31.568382 24230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:31.574456 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.5.98.195:46469)
0924 15:31:31.570604 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.570610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.570754 (+ 144us) server_negotiation.cc:407] Connection header received
0924 15:31:31.570822 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.570824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.570839 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.570931 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:31.571316 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572406 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.573557 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574075 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574246 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.574343 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.574376 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260924 15:31:31.574479 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46469 (local address 127.0.0.1:49358)
0924 15:31:31.570671 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.570681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.570746 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.570948 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.570950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.570953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:31.571011 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.571013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572508 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.572509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.573475 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.573479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574165 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.574241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574254 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.574325 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.574377 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":37,"thread_start_us":82,"threads_started":1}
I20260924 15:31:31.574548 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:44897 (local address 127.0.0.1:34726)
0924 15:31:31.570721 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.570730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.570771 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.571084 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.571085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.571090 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:31.571235 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.571237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572154 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.572155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.573324 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.573326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574269 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.574328 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574332 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.574363 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.574400 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":89,"thread_start_us":62,"threads_started":1}
I20260924 15:31:31.574838 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33085 (local address 127.0.0.1:43092)
0924 15:31:31.570435 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.570481 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.570542 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.571084 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.571085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.571087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:31.571158 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.571159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572524 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.572525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.573607 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.573609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574614 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.574681 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574684 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.574731 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.574766 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":43,"thread_start_us":45,"threads_started":1}
I20260924 15:31:31.574970 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.5.98.194:44897)
0924 15:31:31.570913 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.570918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.570927 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:31:31.570965 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.570967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.570982 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.571148 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:31.571295 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572069 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.573408 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574166 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574683 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574776 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.574800 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.574865 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260924 15:31:31.575158 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.570374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.5.98.193:33085)
0924 15:31:31.570891 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.570914 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.570924 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:31:31.570970 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.570973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.571008 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.571079 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:31.571321 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.572441 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.573690 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.574427 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.574488 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.574577 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.575004 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.575078 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":116,"threads_started":1}
I20260924 15:31:31.575459 24622 tablet_service.cc:1467] Processing CreateTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 (DEFAULT_TABLE table=test-table [id=9aec7840f1294129bc833b423fb27e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:31.575567 24490 tablet_service.cc:1467] Processing CreateTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 (DEFAULT_TABLE table=test-table [id=9aec7840f1294129bc833b423fb27e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:31.575829 24622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:31.575791 24356 tablet_service.cc:1467] Processing CreateTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 (DEFAULT_TABLE table=test-table [id=9aec7840f1294129bc833b423fb27e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:31.576049 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:31.576107 24356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:31.578950 24702 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Bootstrap starting.
I20260924 15:31:31.579324 24704 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Bootstrap starting.
I20260924 15:31:31.579375 24703 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Bootstrap starting.
I20260924 15:31:31.579861 24702 tablet_bootstrap.cc:654] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:31.580000 24704 tablet_bootstrap.cc:654] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:31.580003 24703 tablet_bootstrap.cc:654] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:31.580212 24702 log.cc:824] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:31.580327 24703 log.cc:824] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:31.580327 24704 log.cc:824] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:31.581003 24702 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: No bootstrap required, opened a new log
I20260924 15:31:31.581045 24704 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: No bootstrap required, opened a new log
I20260924 15:31:31.581080 24702 ts_tablet_manager.cc:1397] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:31.581094 24704 ts_tablet_manager.cc:1397] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:31.581248 24703 tablet_bootstrap.cc:492] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: No bootstrap required, opened a new log
I20260924 15:31:31.581337 24703 ts_tablet_manager.cc:1397] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:31:31.583395 24703 raft_consensus.cc:348] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.583545 24703 raft_consensus.cc:374] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:31.583588 24703 raft_consensus.cc:729] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c75fd9c755f491c82966e3b72d663c2, State: Initialized, Role: FOLLOWER
I20260924 15:31:31.583704 24703 consensus_queue.cc:260] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.583618 24702 raft_consensus.cc:348] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.583618 24704 raft_consensus.cc:348] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.583786 24704 raft_consensus.cc:374] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:31.583786 24702 raft_consensus.cc:374] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:31.583823 24702 raft_consensus.cc:729] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44c879a4613472a9bdba59ea5c35fab, State: Initialized, Role: FOLLOWER
I20260924 15:31:31.583823 24704 raft_consensus.cc:729] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb566bc5bb94d7187a41cb837844500, State: Initialized, Role: FOLLOWER
I20260924 15:31:31.583936 24703 ts_tablet_manager.cc:1428] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:31:31.583947 24702 consensus_queue.cc:260] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.584051 24688 heartbeater.cc:503] Master 127.5.98.254:34481 was elected leader, sending a full tablet report...
I20260924 15:31:31.584213 24702 ts_tablet_manager.cc:1428] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:31.584333 24556 heartbeater.cc:503] Master 127.5.98.254:34481 was elected leader, sending a full tablet report...
I20260924 15:31:31.583947 24704 consensus_queue.cc:260] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.584569 24704 ts_tablet_manager.cc:1428] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:31.584872 24424 heartbeater.cc:503] Master 127.5.98.254:34481 was elected leader, sending a full tablet report...
W20260924 15:31:31.668639 24557 tablet.cc:2369] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:31:31.794225 24425 tablet.cc:2369] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:31:31.804529 24689 tablet.cc:2369] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:31.925995 24710 raft_consensus.cc:482] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:31.926148 24710 raft_consensus.cc:504] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.926573 24710 leader_election.cc:288] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d44c879a4613472a9bdba59ea5c35fab (127.5.98.194:44897), 4c75fd9c755f491c82966e3b72d663c2 (127.5.98.195:46469)
I20260924 15:31:31.930471 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.926959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:45919 (local address 127.5.98.194:44897)
0924 15:31:31.927002 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.927005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.927107 (+ 102us) server_negotiation.cc:407] Connection header received
0924 15:31:31.927225 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.927226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.927235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.927290 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:31.927458 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.928436 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.929505 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.930049 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.930144 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.930153 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.930333 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.930382 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:31:31.931015 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.926932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:44897 (local address 127.5.98.193:45919)
0924 15:31:31.927046 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.927056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.927132 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.927299 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.927300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.927302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:31.927396 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.927398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.928553 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.928554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.929385 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.929387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.930145 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.930146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.930225 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.930229 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.930754 (+ 525us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.930801 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260924 15:31:31.931128 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.926839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:53395 (local address 127.5.98.195:46469)
0924 15:31:31.927004 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:31.927006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:31.927088 (+ 82us) server_negotiation.cc:407] Connection header received
0924 15:31:31.927176 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:31.927177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:31.927187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:31.927245 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:31.927572 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.928441 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.930093 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.930734 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.930786 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.930794 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:31.931028 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 15:31:31.931068 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260924 15:31:31.931137 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:31.926715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:46469 (local address 127.5.98.193:53395)
0924 15:31:31.927032 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:31.927043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:31.927118 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:31.927295 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:31.927297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:31.927299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:31.927453 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.927456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.928658 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.928660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.929962 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:31.929966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:31.930853 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:31.930854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:31.930946 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:31.930950 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:31.931018 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:31:31.931054 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":19,"thread_start_us":55,"threads_started":1}
I20260924 15:31:31.931522 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fdde3ef81f4f9883a0fa6eae9bfc55" candidate_uuid: "7bb566bc5bb94d7187a41cb837844500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44c879a4613472a9bdba59ea5c35fab" is_pre_election: true
I20260924 15:31:31.931712 24510 raft_consensus.cc:2454] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb566bc5bb94d7187a41cb837844500 in term 0.
I20260924 15:31:31.931939 24306 leader_election.cc:302] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb566bc5bb94d7187a41cb837844500, d44c879a4613472a9bdba59ea5c35fab; no voters:
I20260924 15:31:31.931958 24642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fdde3ef81f4f9883a0fa6eae9bfc55" candidate_uuid: "7bb566bc5bb94d7187a41cb837844500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c75fd9c755f491c82966e3b72d663c2" is_pre_election: true
I20260924 15:31:31.932094 24642 raft_consensus.cc:2454] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb566bc5bb94d7187a41cb837844500 in term 0.
I20260924 15:31:31.932071 24710 raft_consensus.cc:2781] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:31.932137 24710 raft_consensus.cc:482] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:31.932173 24710 raft_consensus.cc:3037] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:31.933252 24710 raft_consensus.cc:504] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.933416 24710 leader_election.cc:288] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [CANDIDATE]: Term 1 election: Requested vote from peers d44c879a4613472a9bdba59ea5c35fab (127.5.98.194:44897), 4c75fd9c755f491c82966e3b72d663c2 (127.5.98.195:46469)
I20260924 15:31:31.933540 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fdde3ef81f4f9883a0fa6eae9bfc55" candidate_uuid: "7bb566bc5bb94d7187a41cb837844500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d44c879a4613472a9bdba59ea5c35fab"
I20260924 15:31:31.933569 24642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fdde3ef81f4f9883a0fa6eae9bfc55" candidate_uuid: "7bb566bc5bb94d7187a41cb837844500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c75fd9c755f491c82966e3b72d663c2"
I20260924 15:31:31.933611 24510 raft_consensus.cc:3037] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:31.933641 24642 raft_consensus.cc:3037] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:31.934207 24510 raft_consensus.cc:2454] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb566bc5bb94d7187a41cb837844500 in term 1.
I20260924 15:31:31.934351 24306 leader_election.cc:302] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bb566bc5bb94d7187a41cb837844500, d44c879a4613472a9bdba59ea5c35fab; no voters:
I20260924 15:31:31.934419 24710 raft_consensus.cc:2781] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:31.934563 24710 raft_consensus.cc:686] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb566bc5bb94d7187a41cb837844500, State: Running, Role: LEADER
I20260924 15:31:31.934572 24642 raft_consensus.cc:2454] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb566bc5bb94d7187a41cb837844500 in term 1.
I20260924 15:31:31.934648 24710 consensus_queue.cc:237] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } }
I20260924 15:31:31.935506 24230 catalog_manager.cc:5495] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bb566bc5bb94d7187a41cb837844500 (127.5.98.193). New cstate: current_term: 1 leader_uuid: "7bb566bc5bb94d7187a41cb837844500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb566bc5bb94d7187a41cb837844500" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 33085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 } health_report { overall_health: UNKNOWN } } }
I20260924 15:31:32.018302 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:32.015824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.5.98.193:33085)
0924 15:31:32.015890 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:32.015893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:32.015904 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:31:32.015956 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:32.015957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:32.015981 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:32.016064 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:32.016212 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:32.016698 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:32.017540 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:32.017738 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:32.017815 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:32.017956 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:32.018051 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:32.018156 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:32.018206 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 15:31:32.018218 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:31:32.020100 24642 raft_consensus.cc:1261] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 1 FOLLOWER]: Refusing update from remote peer 7bb566bc5bb94d7187a41cb837844500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:32.020102 24510 raft_consensus.cc:1261] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 1 FOLLOWER]: Refusing update from remote peer 7bb566bc5bb94d7187a41cb837844500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:32.020494 24713 consensus_queue.cc:1035] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44c879a4613472a9bdba59ea5c35fab" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:32.020648 24710 consensus_queue.cc:1035] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c75fd9c755f491c82966e3b72d663c2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 46469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:32.022982 24716 mvcc.cc:205] Tried to move back new op lower bound from 7332920901711654912 to 7332920901364998144. Current Snapshot: MvccSnapshot[applied={T|T < 7332920901711654912}]
I20260924 15:31:32.023150 24718 mvcc.cc:205] Tried to move back new op lower bound from 7332920901711654912 to 7332920901364998144. Current Snapshot: MvccSnapshot[applied={T|T < 7332920901711654912}]
I20260924 15:31:32.023473 24717 mvcc.cc:205] Tried to move back new op lower bound from 7332920901711654912 to 7332920901364998144. Current Snapshot: MvccSnapshot[applied={T|T < 7332920901711654912}]
I20260924 15:31:32.025130 24231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:32.025225 24231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:32.025956 24231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ac6ac51f91743ec8004c90fa58bcd65: Sending DeleteTablet for 3 replicas of tablet 94fdde3ef81f4f9883a0fa6eae9bfc55
I20260924 15:31:32.026269 24622 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:32 UTC) from {username='slave'} at 127.0.0.1:49358
I20260924 15:31:32.026356 24356 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:32 UTC) from {username='slave'} at 127.0.0.1:43092
I20260924 15:31:32.026419 24490 tablet_service.cc:1514] Processing DeleteTablet for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:32 UTC) from {username='slave'} at 127.0.0.1:34726
I20260924 15:31:32.026575 24726 tablet_replica.cc:330] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: stopping tablet replica
I20260924 15:31:32.026665 24728 tablet_replica.cc:330] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: stopping tablet replica
I20260924 15:31:32.026690 24726 raft_consensus.cc:2229] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:32.026738 24728 raft_consensus.cc:2229] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:32.026782 24726 raft_consensus.cc:2258] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:32.026839 24728 raft_consensus.cc:2258] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:32.027016 24727 tablet_replica.cc:330] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 7bb566bc5bb94d7187a41cb837844500: stopping tablet replica
I20260924 15:31:32.027043 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24291
I20260924 15:31:32.028334 24728 ts_tablet_manager.cc:1905] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:32.029145 24726 ts_tablet_manager.cc:1905] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:32.029832 24728 ts_tablet_manager.cc:1918] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:32.029922 24728 log.cc:1196] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/94fdde3ef81f4f9883a0fa6eae9bfc55
I20260924 15:31:32.030184 24728 ts_tablet_manager.cc:1939] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P d44c879a4613472a9bdba59ea5c35fab: Deleting consensus metadata
I20260924 15:31:32.030601 24215 catalog_manager.cc:4841] TS d44c879a4613472a9bdba59ea5c35fab (127.5.98.194:44897): tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 (table test-table [id=9aec7840f1294129bc833b423fb27e5f]) successfully deleted
I20260924 15:31:32.030709 24726 ts_tablet_manager.cc:1918] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:32.030772 24726 log.cc:1196] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/94fdde3ef81f4f9883a0fa6eae9bfc55
I20260924 15:31:32.031029 24726 ts_tablet_manager.cc:1939] T 94fdde3ef81f4f9883a0fa6eae9bfc55 P 4c75fd9c755f491c82966e3b72d663c2: Deleting consensus metadata
I20260924 15:31:32.031409 24218 catalog_manager.cc:4841] TS 4c75fd9c755f491c82966e3b72d663c2 (127.5.98.195:46469): tablet 94fdde3ef81f4f9883a0fa6eae9bfc55 (table test-table [id=9aec7840f1294129bc833b423fb27e5f]) successfully deleted
W20260924 15:31:32.033380 24215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:33085 (error 108)
I20260924 15:31:32.033612 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24427
I20260924 15:31:32.033819 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:32.033705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:33085 (local address 127.0.0.1:43120)
0924 15:31:32.033750 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:32.033783 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:31:32.033915 24215 catalog_manager.cc:4584] TS 7bb566bc5bb94d7187a41cb837844500 (127.5.98.193:33085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94fdde3ef81f4f9883a0fa6eae9bfc55: Network error: Client connection negotiation failed: client connection to 127.5.98.193:33085: connect: Connection refused (error 111)
I20260924 15:31:32.038648 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24559
I20260924 15:31:32.042836 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24200
2026-09-24T15:31:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1043 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:32 dist-test-slave-4g6f krb5kdc[24732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:32 dist-test-slave-4g6f krb5kdc[24732](info): set up 2 sockets
Sep 24 15:31:32 dist-test-slave-4g6f krb5kdc[24732](info): commencing operation
krb5kdc: starting...
W20260924 15:31:33.089486 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:33.204519 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43553
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:33.283627 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:33.283905 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:33.283941 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:33.285072 24745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:33.285144 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:33.285171 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:33.285202 24745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:33.285228 24745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:33.286682 24745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:33.286949 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:33.287171 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:33.291962 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.291940 24751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.291944 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.292088 24745 server_base.cc:1034] running on GCE node
I20260924 15:31:33.292357 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:33.292642 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:33.293828 24745 hybrid_clock.cc:648] HybridClock initialized: now 1790263893293812 us; error 51 us; skew 500 ppm
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:33.296231 24745 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:33.296651 24745 webserver.cc:460] Webserver started at http://127.5.98.254:45085/ using document root <none> and password file <none>
I20260924 15:31:33.296867 24745 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:33.296919 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:33.297091 24745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:33.298187 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "1d51072e29954cc3bcd6b6189869dcf5"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "84dc270730f675637ce678cd339c9f5a"
server_key_iv: "a725ce649f4f09975a37adcf498fe67f"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.298550 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "1d51072e29954cc3bcd6b6189869dcf5"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "84dc270730f675637ce678cd339c9f5a"
server_key_iv: "a725ce649f4f09975a37adcf498fe67f"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.299919 24745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:31:33.300611 24761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.300786 24745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:33.300894 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "1d51072e29954cc3bcd6b6189869dcf5"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "84dc270730f675637ce678cd339c9f5a"
server_key_iv: "a725ce649f4f09975a37adcf498fe67f"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.300973 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:33.313622 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:33.315402 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:33.315552 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:33.319398 24745 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:43553
I20260924 15:31:33.319428 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:43553 every 8 connection(s)
I20260924 15:31:33.319823 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:33.320662 24814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:33.322834 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: Bootstrap starting.
I20260924 15:31:33.323349 24814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:33.323603 24814 log.cc:824] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:33.324329 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: No bootstrap required, opened a new log
I20260924 15:31:33.325814 24814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } }
I20260924 15:31:33.325943 24814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:33.325965 24814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d51072e29954cc3bcd6b6189869dcf5, State: Initialized, Role: FOLLOWER
I20260924 15:31:33.326108 24814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } }
I20260924 15:31:33.326191 24814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:33.326213 24814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:33.326241 24814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:33.326709 24814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } }
I20260924 15:31:33.326790 24814 leader_election.cc:302] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d51072e29954cc3bcd6b6189869dcf5; no voters:
I20260924 15:31:33.326898 24814 leader_election.cc:288] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:33.326975 24817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:33.327162 24817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [term 1 LEADER]: Becoming Leader. State: Replica: 1d51072e29954cc3bcd6b6189869dcf5, State: Running, Role: LEADER
I20260924 15:31:33.327185 24814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:33.327251 24817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } }
I20260924 15:31:33.327821 24818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d51072e29954cc3bcd6b6189869dcf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } } }
I20260924 15:31:33.328004 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d51072e29954cc3bcd6b6189869dcf5. Latest consensus state: current_term: 1 leader_uuid: "1d51072e29954cc3bcd6b6189869dcf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d51072e29954cc3bcd6b6189869dcf5" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 43553 } } }
I20260924 15:31:33.328055 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:33.327925 24818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:33.328387 24827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:33.328611 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24745
I20260924 15:31:33.328742 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:33.328915 5515 external_mini_cluster.cc:1428] Setting key aef60d2d1adc5f4956cc52e719b6b570
I20260924 15:31:33.329649 24827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:33.331125 24827 catalog_manager.cc:1347] Generated new cluster ID: d2ae767fee724d9bafc767b28a095bb9
I20260924 15:31:33.331178 24827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:33.340571 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.329391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.5.98.254:43553)
0924 15:31:33.331782 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.331790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.331815 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:31:33.331903 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.331908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.331948 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.332103 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:33.332741 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.333647 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.334400 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.334805 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.334920 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.337260 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:33.337264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:33.337264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:33.337274 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:33.338910 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.339281 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.339284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.339285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.339386 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.339605 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.339606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.339607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.339902 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:33.340069 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.340164 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.340396 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2346,"thread_start_us":76,"threads_started":1}
I20260924 15:31:33.349455 24827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:33.350116 24827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:33.362751 24827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: Generated new TSK 0
I20260924 15:31:33.362908 24827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.98.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
Entry for principal kudu/127.5.98.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab.
I20260924 15:31:33.390997 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:33.471436 24840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:33.471637 24840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:33.471709 24840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:33.472874 24840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:33.472915 24840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:33.472970 24840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:33.474402 24840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.193.keytab
--principal=kudu/127.5.98.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:33.474649 24840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:33.474874 24840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:33.479887 24845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.479904 24846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.479943 24848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.480079 24840 server_base.cc:1034] running on GCE node
I20260924 15:31:33.480317 24840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:33.480568 24840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:33.481732 24840 hybrid_clock.cc:648] HybridClock initialized: now 1790263893481718 us; error 29 us; skew 500 ppm
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:33.484179 24840 init.cc:377] Logged in from keytab as kudu/127.5.98.193@KRBTEST.COM (short username kudu)
I20260924 15:31:33.484551 24840 webserver.cc:460] Webserver started at http://127.5.98.193:45119/ using document root <none> and password file <none>
I20260924 15:31:33.484790 24840 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:33.484836 24840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:33.485006 24840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:33.486056 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "905d000aa6604320b3222bfae8221e9e"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "22a28d42235ce8ed5c66774b13f259f3"
server_key_iv: "6729ecf8f14d854e00c172512e154121"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.486416 24840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "905d000aa6604320b3222bfae8221e9e"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "22a28d42235ce8ed5c66774b13f259f3"
server_key_iv: "6729ecf8f14d854e00c172512e154121"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.487772 24840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:31:33.488520 24855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.488677 24840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:33.488754 24840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "905d000aa6604320b3222bfae8221e9e"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "22a28d42235ce8ed5c66774b13f259f3"
server_key_iv: "6729ecf8f14d854e00c172512e154121"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.488808 24840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:33.510564 24840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:33.512850 24840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:33.512959 24840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:33.513134 24840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:33.513405 24840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:33.513449 24840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.513494 24840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:33.513516 24840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.519361 24840 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:32999
I20260924 15:31:33.519415 24970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:32999 every 8 connection(s)
I20260924 15:31:33.519726 24840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.193@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:33.525681 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24840
I20260924 15:31:33.525802 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:33.526010 5515 external_mini_cluster.cc:1428] Setting key 0888a7680976c2c7764c5d6139d873d9
I20260924 15:31:33.529150 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.520420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:43481 (local address 127.5.98.254:43553)
0924 15:31:33.520496 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.520497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.520703 (+ 206us) server_negotiation.cc:407] Connection header received
0924 15:31:33.521177 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.521178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.521187 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.521244 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:33.522113 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.522605 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.523533 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.523746 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.523794 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.525762 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:33.525764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:33.525765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:33.525772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:33.527503 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.528049 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.528050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.528051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.528114 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.528489 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.528490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.528490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.528596 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:33.528649 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.529022 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.529088 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:31:33.529428 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.520403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43553 (local address 127.5.98.193:43481)
0924 15:31:33.520577 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.520618 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.521096 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.521298 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.521301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.521843 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:33.522033 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.522037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.522684 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.522685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.523461 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.523463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.523805 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.523806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.523929 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.524301 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:33.524309 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:33.525650 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:33.527595 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.527602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.527603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.527978 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.528202 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.528202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.528203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.528407 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.528666 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:33.528672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:33.528834 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.529000 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.529177 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260924 15:31:33.529940 24971 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43553
I20260924 15:31:33.530045 24971 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:33.530228 24971 heartbeater.cc:511] Master 127.5.98.254:43553 requested a full tablet report, sending...
I20260924 15:31:33.530726 24778 ts_manager.cc:194] Registered new tserver with Master: 905d000aa6604320b3222bfae8221e9e (127.5.98.193:32999)
I20260924 15:31:33.531356 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.193@KRBTEST.COM'} at 127.5.98.193:43481
WARNING: no policy specified for kudu/127.5.98.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
Entry for principal kudu/127.5.98.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab.
I20260924 15:31:33.551894 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:33.630501 24976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:33.630681 24976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:33.630707 24976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:33.631847 24976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:33.631886 24976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:33.631911 24976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:33.633343 24976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.194.keytab
--principal=kudu/127.5.98.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:33.633579 24976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:33.633790 24976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:33.638715 24982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.638763 24981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.638732 24976 server_base.cc:1034] running on GCE node
W20260924 15:31:33.638716 24984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.639168 24976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:33.639353 24976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:33.640515 24976 hybrid_clock.cc:648] HybridClock initialized: now 1790263893640497 us; error 63 us; skew 500 ppm
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:33.642956 24976 init.cc:377] Logged in from keytab as kudu/127.5.98.194@KRBTEST.COM (short username kudu)
I20260924 15:31:33.643383 24976 webserver.cc:460] Webserver started at http://127.5.98.194:35911/ using document root <none> and password file <none>
I20260924 15:31:33.643560 24976 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:33.643600 24976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:33.643743 24976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:33.644709 24976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "f3f48cdc405249b88b4c0234c73c0260"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "5450e0089c6ad0d7092081d9a6ac2f70"
server_key_iv: "7e46ff485528404debc5ef7cebdc8fd9"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.645000 24976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "f3f48cdc405249b88b4c0234c73c0260"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "5450e0089c6ad0d7092081d9a6ac2f70"
server_key_iv: "7e46ff485528404debc5ef7cebdc8fd9"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.646222 24976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:33.646919 24991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.647135 24976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:33.647220 24976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "f3f48cdc405249b88b4c0234c73c0260"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "5450e0089c6ad0d7092081d9a6ac2f70"
server_key_iv: "7e46ff485528404debc5ef7cebdc8fd9"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.647274 24976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:33.668228 24976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:33.670468 24976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:33.670578 24976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:33.670753 24976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:33.671064 24976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:33.671099 24976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.671144 24976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:33.671171 24976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.676944 24976 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:40919
I20260924 15:31:33.677006 25105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:40919 every 8 connection(s)
I20260924 15:31:33.677294 24976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.194@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:33.685865 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.678046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:47169 (local address 127.5.98.254:43553)
0924 15:31:33.678103 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.678105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.678330 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:31:33.678801 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.678803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.678813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.678861 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:33.679818 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.680312 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.681307 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.681497 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.681549 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.683522 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:33.683523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:33.683523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:33.683530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:33.684712 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.685084 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.685085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.685086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.685138 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.685351 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.685352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.685352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.685446 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:33.685494 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.685715 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.685774 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:31:33.686013 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 24976
I20260924 15:31:33.686044 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.677940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43553 (local address 127.5.98.194:47169)
0924 15:31:33.678173 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.678229 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.678714 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.678923 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.678927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.679473 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:33.679719 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.679725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.680410 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.680412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.681226 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.681228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.681582 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.681583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.681687 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.682114 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:33.682122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:33.683440 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:33.684783 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.684798 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.684800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.685020 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.685198 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.685199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.685199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.685291 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.685501 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:33.685505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:33.685610 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.685689 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.685827 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260924 15:31:33.686141 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:33.686327 5515 external_mini_cluster.cc:1428] Setting key 7e7aca22b640fafd230aabf38c86055a
I20260924 15:31:33.686522 25106 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43553
I20260924 15:31:33.686610 25106 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:33.686784 25106 heartbeater.cc:511] Master 127.5.98.254:43553 requested a full tablet report, sending...
I20260924 15:31:33.687193 24778 ts_manager.cc:194] Registered new tserver with Master: f3f48cdc405249b88b4c0234c73c0260 (127.5.98.194:40919)
I20260924 15:31:33.687579 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.194@KRBTEST.COM'} at 127.5.98.194:47169
WARNING: no policy specified for kudu/127.5.98.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
Entry for principal kudu/127.5.98.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab.
I20260924 15:31:33.710989 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:33641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:33.794339 25111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:33.794543 25111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:33.794565 25111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:33.795781 25111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:33.795823 25111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:33.795872 25111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:33.797294 25111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:33641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.195.keytab
--principal=kudu/127.5.98.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:43553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:33.797544 25111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:33.797781 25111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:33.802740 25116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:33.802736 25119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.802835 25111 server_base.cc:1034] running on GCE node
W20260924 15:31:33.802768 25117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:33.803184 25111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:33.803404 25111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:33.804558 25111 hybrid_clock.cc:648] HybridClock initialized: now 1790263893804543 us; error 28 us; skew 500 ppm
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:33.807119 25111 init.cc:377] Logged in from keytab as kudu/127.5.98.195@KRBTEST.COM (short username kudu)
I20260924 15:31:33.807644 25111 webserver.cc:460] Webserver started at http://127.5.98.195:33893/ using document root <none> and password file <none>
I20260924 15:31:33.807917 25111 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:33.808012 25111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:33.808151 25111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:33.809224 25111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "3e21f77bcd324809a72362f863264714"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "c3e8a3b2430d4a9f3923d161357d159f"
server_key_iv: "4055923f661ea0def0c167683a1ca344"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.809598 25111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "3e21f77bcd324809a72362f863264714"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "c3e8a3b2430d4a9f3923d161357d159f"
server_key_iv: "4055923f661ea0def0c167683a1ca344"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.810911 25111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:33.811767 25126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.811957 25111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:33.812063 25111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "3e21f77bcd324809a72362f863264714"
format_stamp: "Formatted at 2026-09-24 15:31:33 on dist-test-slave-4g6f"
server_key: "c3e8a3b2430d4a9f3923d161357d159f"
server_key_iv: "4055923f661ea0def0c167683a1ca344"
server_key_version: "encryptionkey@0"
I20260924 15:31:33.812152 25111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:33.832285 25111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:33.834120 25111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:33.834280 25111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:33.834510 25111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:33.834816 25111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:33.834869 25111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.834910 25111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:33.834939 25111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:33.841096 25111 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:41909
I20260924 15:31:33.841138 25240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:41909 every 8 connection(s)
I20260924 15:31:33.841470 25111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:33.845420 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25111
I20260924 15:31:33.845535 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:33.845728 5515 external_mini_cluster.cc:1428] Setting key e9c28998692760b51309fb4b1f573fb5
Sep 24 15:31:33 dist-test-slave-4g6f krb5kdc[24732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263893, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.195@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:33.850606 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.842298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:43195 (local address 127.5.98.254:43553)
0924 15:31:33.842344 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.842347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.842420 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:31:33.842921 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.842922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.842933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.842993 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:33.843954 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.844442 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.845429 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.845713 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.845800 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.848062 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:33.848064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:33.848064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:33.848072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:33.849212 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.849622 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.849623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.849623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.849688 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.849967 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.849968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.849968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.850095 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:33.850166 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.850437 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.850514 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:33.850797 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.842143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:43553 (local address 127.5.98.195:43195)
0924 15:31:33.842306 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.842348 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.842832 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.843047 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.843050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.843604 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:33.843873 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.843879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.844531 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.844532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.845325 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.845327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.845813 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.845814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.845967 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.846559 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:33.846568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:33.847967 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:33.849307 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.849314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.849316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.849534 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.849772 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:33.849773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:33.849773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:33.849886 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:33.850169 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:33.850176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:33.850307 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.850413 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.850574 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260924 15:31:33.851269 25241 heartbeater.cc:347] Connected to a master server at 127.5.98.254:43553
I20260924 15:31:33.851342 25241 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:33.851523 25241 heartbeater.cc:511] Master 127.5.98.254:43553 requested a full tablet report, sending...
I20260924 15:31:33.852051 24778 ts_manager.cc:194] Registered new tserver with Master: 3e21f77bcd324809a72362f863264714 (127.5.98.195:41909)
I20260924 15:31:33.852413 24778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.98.195@KRBTEST.COM'} at 127.5.98.195:43195
I20260924 15:31:33.857371 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:33.865371 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.858809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.5.98.254:43553)
0924 15:31:33.858884 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.858886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.859008 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:31:33.859063 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.859064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.859072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.859122 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:33.859834 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.860318 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.861145 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.861331 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.861398 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.862621 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:33.862622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:33.862623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:33.862630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:33.864130 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.864509 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.864511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.864511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.864569 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:33.864766 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:33.864767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:33.864767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:33.864917 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:33.865031 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.865189 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.865270 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:31:33.867066 24778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:33.868732 24778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:33.875511 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.5.98.193:32999)
0924 15:31:33.871547 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.871551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.871808 (+ 257us) server_negotiation.cc:407] Connection header received
0924 15:31:33.871927 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.871931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.871947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.872017 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:33.872203 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.873210 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.874303 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.875044 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.875124 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.875186 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.875382 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.875428 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260924 15:31:33.875532 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:41909 (local address 127.0.0.1:43176)
0924 15:31:33.871197 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.871257 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.871327 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.871694 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.871698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.871701 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:33.871773 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.871775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.873097 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.873098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.874174 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.874177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.875158 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.875160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.875254 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.875272 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.875357 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.875427 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":31,"thread_start_us":40,"threads_started":1}
I20260924 15:31:33.875532 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:32999 (local address 127.0.0.1:41338)
0924 15:31:33.871739 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.871750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.871805 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.872095 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.872096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.872097 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:33.872137 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.872138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.873311 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.873312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.874197 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.874199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.875160 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.875161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.875254 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.875272 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.875357 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.875427 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260924 15:31:33.875815 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.5.98.195:41909)
0924 15:31:33.871495 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.871500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.871512 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:31:33.871555 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.871558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.871574 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.871710 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:33.871860 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.872991 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.874268 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.875043 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.875124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.875187 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.875683 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.875735 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":88,"threads_started":1}
I20260924 15:31:33.876492 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40919 (local address 127.0.0.1:38340)
0924 15:31:33.871378 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:33.871390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:33.871441 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:33.871702 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:33.871704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:33.871706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:33.871787 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.871789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.872862 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.872863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.874175 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:33.874177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.876256 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.876256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:33.876353 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.876357 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:33.876406 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:31:33.876437 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":80,"threads_started":1}
I20260924 15:31:33.876463 24903 tablet_service.cc:1467] Processing CreateTablet for tablet 53141ab68cf34919937b75709d148f63 (DEFAULT_TABLE table=test-table [id=17f36a8f7b134cf2a5b54cd6eef89aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:33.876488 25175 tablet_service.cc:1467] Processing CreateTablet for tablet 53141ab68cf34919937b75709d148f63 (DEFAULT_TABLE table=test-table [id=17f36a8f7b134cf2a5b54cd6eef89aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:33.876837 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:33.871329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.5.98.194:40919)
0924 15:31:33.871530 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:33.871534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:33.871546 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:31:33.871583 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:33.871584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:33.871596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:33.871923 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:33.872016 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.872756 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.875224 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:33.876093 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:33.876156 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:33.876723 (+ 567us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:33.876746 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:31:33.876781 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260924 15:31:33.876832 24903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53141ab68cf34919937b75709d148f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:33.876830 25175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53141ab68cf34919937b75709d148f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:33.880064 25040 tablet_service.cc:1467] Processing CreateTablet for tablet 53141ab68cf34919937b75709d148f63 (DEFAULT_TABLE table=test-table [id=17f36a8f7b134cf2a5b54cd6eef89aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:33.880292 25258 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Bootstrap starting.
I20260924 15:31:33.880390 25040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53141ab68cf34919937b75709d148f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:33.880728 25259 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Bootstrap starting.
I20260924 15:31:33.880944 25258 tablet_bootstrap.cc:654] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:33.881325 25259 tablet_bootstrap.cc:654] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:33.881556 25258 log.cc:824] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:33.881930 25259 log.cc:824] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:33.882324 25258 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: No bootstrap required, opened a new log
I20260924 15:31:33.882418 25258 ts_tablet_manager.cc:1397] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:31:33.882891 25259 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: No bootstrap required, opened a new log
I20260924 15:31:33.882965 25259 ts_tablet_manager.cc:1397] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:33.883828 25262 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Bootstrap starting.
I20260924 15:31:33.884361 25262 tablet_bootstrap.cc:654] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:33.884470 25258 raft_consensus.cc:348] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.884670 25258 raft_consensus.cc:374] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:33.884707 25258 raft_consensus.cc:729] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 905d000aa6604320b3222bfae8221e9e, State: Initialized, Role: FOLLOWER
I20260924 15:31:33.884848 25258 consensus_queue.cc:260] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.884909 25262 log.cc:824] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:33.884881 25259 raft_consensus.cc:348] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.885053 25259 raft_consensus.cc:374] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:33.885105 25259 raft_consensus.cc:729] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e21f77bcd324809a72362f863264714, State: Initialized, Role: FOLLOWER
I20260924 15:31:33.885150 25258 ts_tablet_manager.cc:1428] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:31:33.885275 24971 heartbeater.cc:503] Master 127.5.98.254:43553 was elected leader, sending a full tablet report...
I20260924 15:31:33.885257 25259 consensus_queue.cc:260] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.885550 25259 ts_tablet_manager.cc:1428] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:31:33.885686 25262 tablet_bootstrap.cc:492] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: No bootstrap required, opened a new log
I20260924 15:31:33.885761 25241 heartbeater.cc:503] Master 127.5.98.254:43553 was elected leader, sending a full tablet report...
I20260924 15:31:33.885785 25262 ts_tablet_manager.cc:1397] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:33.887485 25262 raft_consensus.cc:348] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.887622 25262 raft_consensus.cc:374] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:33.887720 25262 raft_consensus.cc:729] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f48cdc405249b88b4c0234c73c0260, State: Initialized, Role: FOLLOWER
I20260924 15:31:33.887857 25262 consensus_queue.cc:260] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:33.888082 25262 ts_tablet_manager.cc:1428] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:31:33.888152 25106 heartbeater.cc:503] Master 127.5.98.254:43553 was elected leader, sending a full tablet report...
W20260924 15:31:33.927986 25107 tablet.cc:2369] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:31:34.020403 24972 tablet.cc:2369] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:31:34.092036 25242 tablet.cc:2369] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:34.122841 25263 raft_consensus.cc:482] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:34.122996 25263 raft_consensus.cc:504] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:34.123342 25263 leader_election.cc:288] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e21f77bcd324809a72362f863264714 (127.5.98.195:41909), f3f48cdc405249b88b4c0234c73c0260 (127.5.98.194:40919)
I20260924 15:31:34.126398 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.123701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:45969 (local address 127.5.98.194:40919)
0924 15:31:34.123759 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.123763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.123841 (+ 78us) server_negotiation.cc:407] Connection header received
0924 15:31:34.123924 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.123925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.123935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.124003 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.124198 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.124668 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.125667 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.126041 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.126099 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.126129 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.126274 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.126320 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:31:34.126399 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.123611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40919 (local address 127.5.98.193:45969)
0924 15:31:34.123753 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.123764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.123836 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.124023 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.124024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.124026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.124107 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.124126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.124775 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.124776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.125575 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.125577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.126125 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.126126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.126203 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.126207 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.126250 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.126299 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":6,"thread_start_us":81,"threads_started":1}
I20260924 15:31:34.126421 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.123551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:41909 (local address 127.5.98.193:48599)
0924 15:31:34.123729 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.123741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.123825 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.123989 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.123990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.123992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.124072 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.124074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.124729 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.124731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.125572 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.125574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.126152 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.126153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.126226 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.126229 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.126303 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.126338 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":6,"thread_start_us":81,"threads_started":1}
I20260924 15:31:34.126441 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.123631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:48599 (local address 127.5.98.195:41909)
0924 15:31:34.123703 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.123705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.123800 (+ 95us) server_negotiation.cc:407] Connection header received
0924 15:31:34.123903 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.123905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.123916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.123972 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.124153 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.124650 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.125668 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.126048 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.126139 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.126147 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.126323 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.126367 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:31:34.126812 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53141ab68cf34919937b75709d148f63" candidate_uuid: "905d000aa6604320b3222bfae8221e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e21f77bcd324809a72362f863264714" is_pre_election: true
I20260924 15:31:34.126996 25195 raft_consensus.cc:2454] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d000aa6604320b3222bfae8221e9e in term 0.
I20260924 15:31:34.127007 25060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53141ab68cf34919937b75709d148f63" candidate_uuid: "905d000aa6604320b3222bfae8221e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f48cdc405249b88b4c0234c73c0260" is_pre_election: true
I20260924 15:31:34.127166 25060 raft_consensus.cc:2454] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 905d000aa6604320b3222bfae8221e9e in term 0.
I20260924 15:31:34.127264 24856 leader_election.cc:302] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e21f77bcd324809a72362f863264714, 905d000aa6604320b3222bfae8221e9e; no voters:
I20260924 15:31:34.127446 25263 raft_consensus.cc:2781] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:34.127518 25263 raft_consensus.cc:482] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:34.127535 25263 raft_consensus.cc:3037] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.128358 25263 raft_consensus.cc:504] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:34.128501 25263 leader_election.cc:288] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [CANDIDATE]: Term 1 election: Requested vote from peers 3e21f77bcd324809a72362f863264714 (127.5.98.195:41909), f3f48cdc405249b88b4c0234c73c0260 (127.5.98.194:40919)
I20260924 15:31:34.128690 25195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53141ab68cf34919937b75709d148f63" candidate_uuid: "905d000aa6604320b3222bfae8221e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e21f77bcd324809a72362f863264714"
I20260924 15:31:34.128715 25060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53141ab68cf34919937b75709d148f63" candidate_uuid: "905d000aa6604320b3222bfae8221e9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f48cdc405249b88b4c0234c73c0260"
I20260924 15:31:34.128779 25060 raft_consensus.cc:3037] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.128779 25195 raft_consensus.cc:3037] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.129659 25195 raft_consensus.cc:2454] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d000aa6604320b3222bfae8221e9e in term 1.
I20260924 15:31:34.129662 25060 raft_consensus.cc:2454] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 905d000aa6604320b3222bfae8221e9e in term 1.
I20260924 15:31:34.129843 24858 leader_election.cc:302] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 905d000aa6604320b3222bfae8221e9e, f3f48cdc405249b88b4c0234c73c0260; no voters:
I20260924 15:31:34.129989 25263 raft_consensus.cc:2781] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:34.130146 25263 raft_consensus.cc:686] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [term 1 LEADER]: Becoming Leader. State: Replica: 905d000aa6604320b3222bfae8221e9e, State: Running, Role: LEADER
I20260924 15:31:34.130252 25263 consensus_queue.cc:237] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } }
I20260924 15:31:34.131096 24778 catalog_manager.cc:5495] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 905d000aa6604320b3222bfae8221e9e (127.5.98.193). New cstate: current_term: 1 leader_uuid: "905d000aa6604320b3222bfae8221e9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "905d000aa6604320b3222bfae8221e9e" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 32999 } health_report { overall_health: HEALTHY } } }
I20260924 15:31:34.156253 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.153933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.5.98.193:32999)
0924 15:31:34.153987 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.153989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.153996 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:31:34.154040 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.154041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.154079 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.154152 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:34.154282 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.154749 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.155480 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.155710 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.155773 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.155908 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:34.156012 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:34.156076 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.156180 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.156197 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:31:34.158072 25195 raft_consensus.cc:1261] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 1 FOLLOWER]: Refusing update from remote peer 905d000aa6604320b3222bfae8221e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:34.158154 25060 raft_consensus.cc:1261] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 1 FOLLOWER]: Refusing update from remote peer 905d000aa6604320b3222bfae8221e9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:34.158496 25263 consensus_queue.cc:1035] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e21f77bcd324809a72362f863264714" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:34.158648 25269 consensus_queue.cc:1035] T 53141ab68cf34919937b75709d148f63 P 905d000aa6604320b3222bfae8221e9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f48cdc405249b88b4c0234c73c0260" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:34.160907 25272 mvcc.cc:205] Tried to move back new op lower bound from 7332920910468927488 to 7332920910358188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332920910468927488}]
I20260924 15:31:34.161506 25273 mvcc.cc:205] Tried to move back new op lower bound from 7332920910468927488 to 7332920910358188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332920910468927488}]
I20260924 15:31:34.161743 25274 mvcc.cc:205] Tried to move back new op lower bound from 7332920910468927488 to 7332920910358188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332920910468927488 or (T in {7332920910468927488})}]
I20260924 15:31:34.163141 24778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:34.163254 24778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:34.164031 24778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d51072e29954cc3bcd6b6189869dcf5: Sending DeleteTablet for 3 replicas of tablet 53141ab68cf34919937b75709d148f63
I20260924 15:31:34.164373 25040 tablet_service.cc:1514] Processing DeleteTablet for tablet 53141ab68cf34919937b75709d148f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:34 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:38340
I20260924 15:31:34.164633 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24840
I20260924 15:31:34.164649 25175 tablet_service.cc:1514] Processing DeleteTablet for tablet 53141ab68cf34919937b75709d148f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:34 UTC) from {username='kudu', principal='kudu/127.5.98.254@KRBTEST.COM'} at 127.0.0.1:43176
I20260924 15:31:34.165334 25282 tablet_replica.cc:330] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: stopping tablet replica
I20260924 15:31:34.165436 25282 raft_consensus.cc:2229] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:34.165540 25282 raft_consensus.cc:2258] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:34.165967 25283 tablet_replica.cc:330] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: stopping tablet replica
I20260924 15:31:34.166016 25282 ts_tablet_manager.cc:1905] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:34.166062 25283 raft_consensus.cc:2229] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:34.166157 25283 raft_consensus.cc:2258] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:34.166616 25283 ts_tablet_manager.cc:1905] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:34.168329 25283 ts_tablet_manager.cc:1918] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:34.168390 25283 log.cc:1196] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/53141ab68cf34919937b75709d148f63
I20260924 15:31:34.168646 25283 ts_tablet_manager.cc:1939] T 53141ab68cf34919937b75709d148f63 P 3e21f77bcd324809a72362f863264714: Deleting consensus metadata
I20260924 15:31:34.169123 24762 catalog_manager.cc:4841] TS 3e21f77bcd324809a72362f863264714 (127.5.98.195:41909): tablet 53141ab68cf34919937b75709d148f63 (table test-table [id=17f36a8f7b134cf2a5b54cd6eef89aa5]) successfully deleted
W20260924 15:31:34.171804 24762 connection.cc:535] client connection to 127.5.98.193:32999 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:31:34.171839 25282 ts_tablet_manager.cc:1918] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:34.171908 25282 log.cc:1196] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/53141ab68cf34919937b75709d148f63
W20260924 15:31:34.171922 24762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:31:34.172048 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24976
I20260924 15:31:34.172161 25282 ts_tablet_manager.cc:1939] T 53141ab68cf34919937b75709d148f63 P f3f48cdc405249b88b4c0234c73c0260: Deleting consensus metadata
I20260924 15:31:34.172361 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.172250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:32999 (local address 127.0.0.1:41360)
0924 15:31:34.172294 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.172324 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:32999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 15:31:34.172468 24762 catalog_manager.cc:4584] TS 905d000aa6604320b3222bfae8221e9e (127.5.98.193:32999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53141ab68cf34919937b75709d148f63: Network error: Client connection negotiation failed: client connection to 127.5.98.193:32999: connect: Connection refused (error 111)
I20260924 15:31:34.177793 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.177677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:40919 (local address 127.0.0.1:38342)
0924 15:31:34.177730 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.177758 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.194:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:31:34.177862 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25111
I20260924 15:31:34.182502 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 24745
2026-09-24T15:31:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T15:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:34Z Disabled control of system clock
I20260924 15:31:34.210650 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:35791
--webserver_interface=127.5.98.254
--webserver_port=0
--builtin_ntp_servers=127.5.98.212:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:35791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:34.290453 25288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:34.290656 25288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:34.290676 25288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:34.291767 25288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:34.291800 25288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:34.291813 25288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:34.291826 25288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:34.291837 25288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:34.293105 25288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:35791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:34.293295 25288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:34.293505 25288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:34.298406 25293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.298426 25294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.298406 25296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:34.298434 25288 server_base.cc:1034] running on GCE node
I20260924 15:31:34.298894 25288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:34.299185 25288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:34.300351 25288 hybrid_clock.cc:648] HybridClock initialized: now 1790263894300337 us; error 40 us; skew 500 ppm
I20260924 15:31:34.301497 25288 webserver.cc:460] Webserver started at http://127.5.98.254:36919/ using document root <none> and password file <none>
I20260924 15:31:34.301731 25288 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:34.301803 25288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:34.301955 25288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:34.303056 25288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "743222586333499e8a0397c2e001f9ec"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "25b0854b721e977dd82ca1824f353f1e"
server_key_iv: "68a9af29b740159af80f6a4b2641b707"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.303417 25288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "743222586333499e8a0397c2e001f9ec"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "25b0854b721e977dd82ca1824f353f1e"
server_key_iv: "68a9af29b740159af80f6a4b2641b707"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.304814 25288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:34.305554 25303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.305756 25288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:34.305867 25288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "743222586333499e8a0397c2e001f9ec"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "25b0854b721e977dd82ca1824f353f1e"
server_key_iv: "68a9af29b740159af80f6a4b2641b707"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.305958 25288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:34.322690 25288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:34.323078 25288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:34.323233 25288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:34.327093 25288 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:35791
I20260924 15:31:34.327126 25355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:35791 every 8 connection(s)
I20260924 15:31:34.327487 25288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:34.328508 25356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:34.330705 25356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: Bootstrap starting.
I20260924 15:31:34.331311 25356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:34.331616 25356 log.cc:824] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:34.332311 25356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: No bootstrap required, opened a new log
I20260924 15:31:34.333878 25356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } }
I20260924 15:31:34.334010 25356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:34.334031 25356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743222586333499e8a0397c2e001f9ec, State: Initialized, Role: FOLLOWER
I20260924 15:31:34.334196 25356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } }
I20260924 15:31:34.334259 25356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:34.334280 25356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:34.334340 25356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.334739 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25288
I20260924 15:31:34.334817 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:34.334925 25356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } }
I20260924 15:31:34.334996 5515 external_mini_cluster.cc:1428] Setting key 0f9aaf615834bd57f2068ba8651f1534
I20260924 15:31:34.335047 25356 leader_election.cc:302] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 743222586333499e8a0397c2e001f9ec; no voters:
I20260924 15:31:34.335230 25356 leader_election.cc:288] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:34.335261 25359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:34.335381 25359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [term 1 LEADER]: Becoming Leader. State: Replica: 743222586333499e8a0397c2e001f9ec, State: Running, Role: LEADER
I20260924 15:31:34.335487 25356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:34.335466 25359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } }
I20260924 15:31:34.336201 25360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 743222586333499e8a0397c2e001f9ec. Latest consensus state: current_term: 1 leader_uuid: "743222586333499e8a0397c2e001f9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } } }
I20260924 15:31:34.336297 25360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:34.336300 25361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "743222586333499e8a0397c2e001f9ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "743222586333499e8a0397c2e001f9ec" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 35791 } } }
I20260924 15:31:34.336441 25361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec [sys.catalog]: This master's current role is: LEADER
W20260924 15:31:34.337743 25377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:31:34.337806 25377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:31:34.337857 25368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:34.338034 25368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:34.339468 25368 catalog_manager.cc:1347] Generated new cluster ID: 4096cc5930f24505994b018f68069d95
I20260924 15:31:34.339517 25368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:34.342082 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.335848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.5.98.254:35791)
0924 15:31:34.336790 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.336800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.336840 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:31:34.336981 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.336990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.337071 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.337223 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:34.337495 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.338887 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.339985 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.340589 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.340720 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.341367 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:34.341371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:34.341371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:34.341577 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:34.341620 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:34.341634 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:34.341707 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.341775 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.341863 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":248,"threads_started":1}
I20260924 15:31:34.348999 25368 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:34.349437 25368 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:34.354712 25368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: Generated new TSK 0
I20260924 15:31:34.354895 25368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:34.360101 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.193:0
--local_ip_for_outbound_sockets=127.5.98.193
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--builtin_ntp_servers=127.5.98.212:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:34.440528 25380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:34.440716 25380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:34.440737 25380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:34.441864 25380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:34.441901 25380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:34.441926 25380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.193
I20260924 15:31:34.443225 25380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.98.193
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.193
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:34.443476 25380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:34.443828 25380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:34.449538 25385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.449635 25386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.449684 25388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:34.449851 25380 server_base.cc:1034] running on GCE node
I20260924 15:31:34.450062 25380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:34.450294 25380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:34.451450 25380 hybrid_clock.cc:648] HybridClock initialized: now 1790263894451439 us; error 30 us; skew 500 ppm
I20260924 15:31:34.452692 25380 webserver.cc:460] Webserver started at http://127.5.98.193:36221/ using document root <none> and password file <none>
I20260924 15:31:34.452950 25380 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:34.453017 25380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:34.453164 25380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:34.454245 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data/instance:
uuid: "381872cd1a4748339459c8b5ba883647"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "4a0d78dfcf5e3eaaf86f273e33bfd5d6"
server_key_iv: "1f71400f2d64efe44edb5ed29f1d04aa"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.454640 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance:
uuid: "381872cd1a4748339459c8b5ba883647"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "4a0d78dfcf5e3eaaf86f273e33bfd5d6"
server_key_iv: "1f71400f2d64efe44edb5ed29f1d04aa"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.456001 25380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:34.456851 25394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.457028 25380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:34.457125 25380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal
uuid: "381872cd1a4748339459c8b5ba883647"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "4a0d78dfcf5e3eaaf86f273e33bfd5d6"
server_key_iv: "1f71400f2d64efe44edb5ed29f1d04aa"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.457203 25380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:34.474671 25380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:34.474967 25380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:34.475131 25380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:34.475358 25380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:34.475718 25380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:34.475780 25380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.475821 25380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:34.475849 25380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.481530 25380 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.193:39607
I20260924 15:31:34.481575 25508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.193:39607 every 8 connection(s)
I20260924 15:31:34.481882 25380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/data/info.pb
I20260924 15:31:34.485045 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25380
I20260924 15:31:34.485126 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-0/wal/instance
I20260924 15:31:34.485299 5515 external_mini_cluster.cc:1428] Setting key 602752f5e5741480d2450d141995fffc
I20260924 15:31:34.486253 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.194:0
--local_ip_for_outbound_sockets=127.5.98.194
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--builtin_ntp_servers=127.5.98.212:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:31:34.487082 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.482524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.193:45019 (local address 127.5.98.254:35791)
0924 15:31:34.482589 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.482591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.482835 (+ 244us) server_negotiation.cc:407] Connection header received
0924 15:31:34.483299 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.483300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.483312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.483366 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:34.484098 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.484558 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.485609 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.485865 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.485944 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.486822 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:34.486824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:34.486824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:34.486828 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:34.486841 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:34.486844 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:34.486905 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.486980 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.487007 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:31:34.487278 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.482533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:35791 (local address 127.5.98.193:45019)
0924 15:31:34.482728 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.482759 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.483211 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.483450 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.483453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.483462 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:34.484027 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.484032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.484633 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.484633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.485529 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.485532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.485965 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.485967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.486115 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.486670 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:34.486670 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:34.486697 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:34.486718 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:34.486908 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:34.486913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:34.486916 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.486974 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.487068 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260924 15:31:34.487805 25509 heartbeater.cc:347] Connected to a master server at 127.5.98.254:35791
I20260924 15:31:34.487886 25509 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:34.488070 25509 heartbeater.cc:511] Master 127.5.98.254:35791 requested a full tablet report, sending...
I20260924 15:31:34.488688 25320 ts_manager.cc:194] Registered new tserver with Master: 381872cd1a4748339459c8b5ba883647 (127.5.98.193:39607)
I20260924 15:31:34.489621 25320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.193:45019
W20260924 15:31:34.566565 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:34.566740 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:34.566759 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:34.567838 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:34.567871 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:34.567895 25512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.194
I20260924 15:31:34.569126 25512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.98.194
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.194
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:34.569293 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:34.569545 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:34.574656 25520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.574669 25518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.574708 25517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:34.574946 25512 server_base.cc:1034] running on GCE node
I20260924 15:31:34.575174 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:34.575376 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:34.576517 25512 hybrid_clock.cc:648] HybridClock initialized: now 1790263894576505 us; error 27 us; skew 500 ppm
I20260924 15:31:34.577859 25512 webserver.cc:460] Webserver started at http://127.5.98.194:42797/ using document root <none> and password file <none>
I20260924 15:31:34.578104 25512 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:34.578152 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:34.578291 25512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:34.579391 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data/instance:
uuid: "e01b9f5f2178411ab9a117a65b635360"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "26653c5acef87a76f23004f58f4e5717"
server_key_iv: "0d0a0ecb91ce31e3c738c4f355e7ff57"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.579821 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance:
uuid: "e01b9f5f2178411ab9a117a65b635360"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "26653c5acef87a76f23004f58f4e5717"
server_key_iv: "0d0a0ecb91ce31e3c738c4f355e7ff57"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.581231 25512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:34.581995 25526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.582219 25512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:34.582295 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal
uuid: "e01b9f5f2178411ab9a117a65b635360"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "26653c5acef87a76f23004f58f4e5717"
server_key_iv: "0d0a0ecb91ce31e3c738c4f355e7ff57"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.582396 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:34.596194 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:34.596464 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:34.596606 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:34.596817 25512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:34.597121 25512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:34.597174 25512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.597213 25512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:34.597239 25512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.602803 25512 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.194:38471
I20260924 15:31:34.602850 25642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.194:38471 every 8 connection(s)
I20260924 15:31:34.603142 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/data/info.pb
I20260924 15:31:34.607995 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.603894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:47963 (local address 127.5.98.254:35791)
0924 15:31:34.603945 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.603948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.604171 (+ 223us) server_negotiation.cc:407] Connection header received
0924 15:31:34.604606 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.604607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.604618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.604672 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:34.605270 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.605749 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.606685 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.606869 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.606922 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.607776 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:34.607777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:34.607778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:34.607782 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:34.607793 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:34.607796 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:34.607836 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.607899 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.607920 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:31:34.608248 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.603888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:35791 (local address 127.5.98.194:47963)
0924 15:31:34.604061 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.604093 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.604523 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.604722 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.604725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.604733 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:34.605203 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.605207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.605827 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.605827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.606586 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.606588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.606945 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.606945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.607083 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.607607 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:34.607608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:34.607636 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:34.607697 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:34.607836 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:34.607839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:34.607841 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.607895 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.607990 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260924 15:31:34.608759 25643 heartbeater.cc:347] Connected to a master server at 127.5.98.254:35791
I20260924 15:31:34.608856 25643 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:34.609022 25643 heartbeater.cc:511] Master 127.5.98.254:35791 requested a full tablet report, sending...
I20260924 15:31:34.609421 25320 ts_manager.cc:194] Registered new tserver with Master: e01b9f5f2178411ab9a117a65b635360 (127.5.98.194:38471)
I20260924 15:31:34.609778 25320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.194:47963
I20260924 15:31:34.611443 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25512
I20260924 15:31:34.611547 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal/instance
I20260924 15:31:34.611770 5515 external_mini_cluster.cc:1428] Setting key 0c4f1670e4d2505cd81a2edfa5647d3d
I20260924 15:31:34.612576 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.98.195:0
--local_ip_for_outbound_sockets=127.5.98.195
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--builtin_ntp_servers=127.5.98.212:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:31:34.692418 25646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:34.692597 25646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:34.692617 25646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:34.693704 25646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:34.693739 25646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:34.693763 25646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.98.195
I20260924 15:31:34.695125 25646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.98.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.98.195
--webserver_port=0
--tserver_master_addrs=127.5.98.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.98.195
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:34.695362 25646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:34.695647 25646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:31:34.700414 25646 server_base.cc:1034] running on GCE node
W20260924 15:31:34.700383 25654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.700383 25652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:34.700394 25651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:34.700865 25646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:34.701100 25646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:34.702262 25646 hybrid_clock.cc:648] HybridClock initialized: now 1790263894702251 us; error 28 us; skew 500 ppm
I20260924 15:31:34.703480 25646 webserver.cc:460] Webserver started at http://127.5.98.195:35253/ using document root <none> and password file <none>
I20260924 15:31:34.703783 25646 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:34.703851 25646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:34.703994 25646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:34.704988 25646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data/instance:
uuid: "9addd93b3c2d4f879838cc0a54df4bb2"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "9e3b6711ebe758e02cefc6f223669f32"
server_key_iv: "8ed3a6e7ea12e700ceeade01f5446169"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.705336 25646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance:
uuid: "9addd93b3c2d4f879838cc0a54df4bb2"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "9e3b6711ebe758e02cefc6f223669f32"
server_key_iv: "8ed3a6e7ea12e700ceeade01f5446169"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.706581 25646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:34.707261 25660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.707429 25646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:34.707521 25646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal
uuid: "9addd93b3c2d4f879838cc0a54df4bb2"
format_stamp: "Formatted at 2026-09-24 15:31:34 on dist-test-slave-4g6f"
server_key: "9e3b6711ebe758e02cefc6f223669f32"
server_key_iv: "8ed3a6e7ea12e700ceeade01f5446169"
server_key_version: "encryptionkey@0"
I20260924 15:31:34.707607 25646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:34.739611 25646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:34.739916 25646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:34.740059 25646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:34.740286 25646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:31:34.740595 25646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:31:34.740650 25646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.740690 25646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:31:34.740718 25646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:34.746582 25646 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.195:45777
I20260924 15:31:34.746655 25773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.195:45777 every 8 connection(s)
I20260924 15:31:34.746968 25646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/data/info.pb
I20260924 15:31:34.747351 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25646
I20260924 15:31:34.747467 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal/instance
I20260924 15:31:34.747716 5515 external_mini_cluster.cc:1428] Setting key b4114d3bc1cd72ca06c5ecd8094cb518
I20260924 15:31:34.752117 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.747783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.195:44287 (local address 127.5.98.254:35791)
0924 15:31:34.747880 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.747883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.748106 (+ 223us) server_negotiation.cc:407] Connection header received
0924 15:31:34.748747 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.748747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.748760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.748810 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:34.749445 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.749923 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.750838 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.751026 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.751081 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.751894 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:34.751895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:34.751895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:34.751900 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:34.751916 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:34.751920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:34.751966 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.752036 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.752061 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 15:31:34.752357 25777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.747813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:35791 (local address 127.5.98.195:44287)
0924 15:31:34.747999 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.748043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.748607 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.748876 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.748879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.748887 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:34.749360 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.749363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.749998 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.749998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.750760 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.750762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.751101 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.751102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.751205 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.751764 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:31:34.751764 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:34.751791 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:31:34.751813 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:34.751980 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:34.751984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:34.751986 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.752038 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.752114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 15:31:34.752804 25774 heartbeater.cc:347] Connected to a master server at 127.5.98.254:35791
I20260924 15:31:34.752878 25774 heartbeater.cc:464] Registering TS with master...
I20260924 15:31:34.753017 25774 heartbeater.cc:511] Master 127.5.98.254:35791 requested a full tablet report, sending...
I20260924 15:31:34.753294 25320 ts_manager.cc:194] Registered new tserver with Master: 9addd93b3c2d4f879838cc0a54df4bb2 (127.5.98.195:45777)
I20260924 15:31:34.753683 25320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.98.195:44287
I20260924 15:31:34.759539 5515 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:31:34.763433 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.760932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.5.98.254:35791)
0924 15:31:34.761002 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.761004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.761058 (+ 54us) server_negotiation.cc:407] Connection header received
0924 15:31:34.761106 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.761106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.761116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.761173 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:34.761294 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.761751 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.762443 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.762611 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.762658 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.763213 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:34.763215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:34.763215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:34.763218 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:34.763230 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:34.763234 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:34.763277 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.763326 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.763346 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:31:34.764977 25320 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:31:34.766479 25320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:31:34.772727 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.768815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.5.98.194:38471)
0924 15:31:34.769107 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.769113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.769122 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:31:34.769159 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.769161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.769177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.769253 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.769463 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770574 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.771829 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772319 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772422 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.772522 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.772608 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.772645 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":73,"threads_started":1}
I20260924 15:31:34.772716 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.768634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.194:38471 (local address 127.0.0.1:39260)
0924 15:31:34.768811 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.768849 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.768912 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.769266 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.769267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.769269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.769354 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.769356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770663 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.770664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.771767 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.771769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772426 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.772504 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.772517 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.772593 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.772646 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260924 15:31:34.772818 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.768634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:39607 (local address 127.0.0.1:52172)
0924 15:31:34.768916 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.768927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.768986 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.769208 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.769211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.769214 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.769280 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.769282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770372 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.770373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.771651 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.771702 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772596 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.772659 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.772663 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.772693 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.772732 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":30,"thread_start_us":100,"threads_started":1}
I20260924 15:31:34.773132 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.768848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:45777 (local address 127.0.0.1:39114)
0924 15:31:34.769346 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.769357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.769403 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.769660 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.769660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.769662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.769697 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.769699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770916 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.770918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.771968 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.771971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772923 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.772984 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.772988 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.773037 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.773071 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":92,"threads_started":1}
I20260924 15:31:34.773229 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.768715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.5.98.193:39607)
0924 15:31:34.769016 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.769021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.769031 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:31:34.769072 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.769074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.769090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.769370 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.769437 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770280 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.771782 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772491 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772980 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.773075 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.773100 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.773137 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
I20260924 15:31:34.773350 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.769049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.5.98.195:45777)
0924 15:31:34.769205 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.769209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.769476 (+ 267us) server_negotiation.cc:407] Connection header received
0924 15:31:34.769517 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.769519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.769533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.769600 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.769783 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.770798 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772058 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.772735 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.772791 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.772870 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.773230 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.773271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260924 15:31:34.773778 25576 tablet_service.cc:1467] Processing CreateTablet for tablet 151e46d91b684057abba75ccbdd5ba60 (DEFAULT_TABLE table=test-table [id=33cfe77ee1994413a9ec5d6cd17e6ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:34.774055 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151e46d91b684057abba75ccbdd5ba60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:34.773973 25708 tablet_service.cc:1467] Processing CreateTablet for tablet 151e46d91b684057abba75ccbdd5ba60 (DEFAULT_TABLE table=test-table [id=33cfe77ee1994413a9ec5d6cd17e6ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:34.773964 25442 tablet_service.cc:1467] Processing CreateTablet for tablet 151e46d91b684057abba75ccbdd5ba60 (DEFAULT_TABLE table=test-table [id=33cfe77ee1994413a9ec5d6cd17e6ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:31:34.774291 25708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151e46d91b684057abba75ccbdd5ba60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:34.774291 25442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 151e46d91b684057abba75ccbdd5ba60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:34.776661 25789 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Bootstrap starting.
I20260924 15:31:34.777171 25789 tablet_bootstrap.cc:654] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:34.777428 25789 log.cc:824] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:34.777534 25790 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: Bootstrap starting.
I20260924 15:31:34.777937 25791 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Bootstrap starting.
I20260924 15:31:34.778153 25790 tablet_bootstrap.cc:654] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:34.778460 25790 log.cc:824] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:34.778555 25791 tablet_bootstrap.cc:654] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:34.778872 25791 log.cc:824] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:34.779260 25790 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: No bootstrap required, opened a new log
I20260924 15:31:34.779345 25790 ts_tablet_manager.cc:1397] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:31:34.779722 25791 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: No bootstrap required, opened a new log
I20260924 15:31:34.779800 25791 ts_tablet_manager.cc:1397] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:31:34.779888 25789 tablet_bootstrap.cc:492] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: No bootstrap required, opened a new log
I20260924 15:31:34.780025 25789 ts_tablet_manager.cc:1397] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:31:34.781458 25791 raft_consensus.cc:348] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.781611 25791 raft_consensus.cc:374] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:34.781632 25791 raft_consensus.cc:729] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9addd93b3c2d4f879838cc0a54df4bb2, State: Initialized, Role: FOLLOWER
I20260924 15:31:34.781749 25791 consensus_queue.cc:260] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.782019 25791 ts_tablet_manager.cc:1428] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:31:34.782051 25774 heartbeater.cc:503] Master 127.5.98.254:35791 was elected leader, sending a full tablet report...
I20260924 15:31:34.781978 25790 raft_consensus.cc:348] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.782132 25790 raft_consensus.cc:374] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:34.782177 25790 raft_consensus.cc:729] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381872cd1a4748339459c8b5ba883647, State: Initialized, Role: FOLLOWER
I20260924 15:31:34.782312 25790 consensus_queue.cc:260] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.782586 25790 ts_tablet_manager.cc:1428] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 15:31:34.782610 25509 heartbeater.cc:503] Master 127.5.98.254:35791 was elected leader, sending a full tablet report...
I20260924 15:31:34.783304 25789 raft_consensus.cc:348] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.783471 25789 raft_consensus.cc:374] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:34.783512 25789 raft_consensus.cc:729] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01b9f5f2178411ab9a117a65b635360, State: Initialized, Role: FOLLOWER
I20260924 15:31:34.783619 25789 consensus_queue.cc:260] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.783878 25789 ts_tablet_manager.cc:1428] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260924 15:31:34.783952 25643 heartbeater.cc:503] Master 127.5.98.254:35791 was elected leader, sending a full tablet report...
W20260924 15:31:34.853715 25644 tablet.cc:2369] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:31:34.863138 25797 raft_consensus.cc:482] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:31:34.863291 25797 raft_consensus.cc:504] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.863729 25797 leader_election.cc:288] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 381872cd1a4748339459c8b5ba883647 (127.5.98.193:39607), 9addd93b3c2d4f879838cc0a54df4bb2 (127.5.98.195:45777)
I20260924 15:31:34.866559 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.863882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:39607 (local address 127.5.98.194:35953)
0924 15:31:34.863945 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.863957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.863999 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.864183 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.864185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.864187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.864267 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.864269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.865001 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.865002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.865762 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.865764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.866311 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.866312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.866381 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.866384 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.866441 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.866473 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:31:34.866573 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.863979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:35953 (local address 127.5.98.193:39607)
0924 15:31:34.864022 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.864024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.864045 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:31:34.864072 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.864073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.864084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.864140 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.864356 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.864914 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.865861 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.866233 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.866288 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.866294 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.866455 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.866501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:31:34.866680 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.864007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.98.194:44921 (local address 127.5.98.195:45777)
0924 15:31:34.864062 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.864064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.864191 (+ 127us) server_negotiation.cc:407] Connection header received
0924 15:31:34.864243 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.864244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.864255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.864319 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:31:34.864502 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.864974 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.865906 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.866253 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.866346 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.866353 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.866548 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.866595 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:31:34.866688 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.863944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.195:45777 (local address 127.5.98.194:44921)
0924 15:31:34.864106 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.864115 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:34.864163 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:34.864345 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:34.864346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:34.864348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:31:34.864428 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.864430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.865061 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.865061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.865791 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:34.865792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.866372 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.866373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:34.866471 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.866475 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:31:34.866531 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:31:34.866566 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":9,"thread_start_us":91,"threads_started":1}
I20260924 15:31:34.867081 25462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151e46d91b684057abba75ccbdd5ba60" candidate_uuid: "e01b9f5f2178411ab9a117a65b635360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381872cd1a4748339459c8b5ba883647" is_pre_election: true
I20260924 15:31:34.867096 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151e46d91b684057abba75ccbdd5ba60" candidate_uuid: "e01b9f5f2178411ab9a117a65b635360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" is_pre_election: true
I20260924 15:31:34.867234 25728 raft_consensus.cc:2454] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01b9f5f2178411ab9a117a65b635360 in term 0.
I20260924 15:31:34.867237 25462 raft_consensus.cc:2454] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01b9f5f2178411ab9a117a65b635360 in term 0.
I20260924 15:31:34.867478 25529 leader_election.cc:302] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381872cd1a4748339459c8b5ba883647, e01b9f5f2178411ab9a117a65b635360; no voters:
I20260924 15:31:34.867599 25797 raft_consensus.cc:2781] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:31:34.867671 25797 raft_consensus.cc:482] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:31:34.867698 25797 raft_consensus.cc:3037] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.868397 25797 raft_consensus.cc:504] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.868543 25797 leader_election.cc:288] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [CANDIDATE]: Term 1 election: Requested vote from peers 381872cd1a4748339459c8b5ba883647 (127.5.98.193:39607), 9addd93b3c2d4f879838cc0a54df4bb2 (127.5.98.195:45777)
I20260924 15:31:34.868690 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151e46d91b684057abba75ccbdd5ba60" candidate_uuid: "e01b9f5f2178411ab9a117a65b635360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9addd93b3c2d4f879838cc0a54df4bb2"
I20260924 15:31:34.868695 25462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "151e46d91b684057abba75ccbdd5ba60" candidate_uuid: "e01b9f5f2178411ab9a117a65b635360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381872cd1a4748339459c8b5ba883647"
I20260924 15:31:34.868775 25728 raft_consensus.cc:3037] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.868767 25462 raft_consensus.cc:3037] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:34.869477 25728 raft_consensus.cc:2454] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01b9f5f2178411ab9a117a65b635360 in term 1.
I20260924 15:31:34.869491 25462 raft_consensus.cc:2454] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01b9f5f2178411ab9a117a65b635360 in term 1.
I20260924 15:31:34.869657 25530 leader_election.cc:302] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9addd93b3c2d4f879838cc0a54df4bb2, e01b9f5f2178411ab9a117a65b635360; no voters:
I20260924 15:31:34.869760 25797 raft_consensus.cc:2781] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:34.869926 25797 raft_consensus.cc:686] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 1 LEADER]: Becoming Leader. State: Replica: e01b9f5f2178411ab9a117a65b635360, State: Running, Role: LEADER
I20260924 15:31:34.870028 25797 consensus_queue.cc:237] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } }
I20260924 15:31:34.870846 25320 catalog_manager.cc:5495] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 reported cstate change: term changed from 0 to 1, leader changed from <none> to e01b9f5f2178411ab9a117a65b635360 (127.5.98.194). New cstate: current_term: 1 leader_uuid: "e01b9f5f2178411ab9a117a65b635360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01b9f5f2178411ab9a117a65b635360" member_type: VOTER last_known_addr { host: "127.5.98.194" port: 38471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 } health_report { overall_health: UNKNOWN } } }
I20260924 15:31:34.889369 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.886958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.5.98.194:38471)
0924 15:31:34.887021 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:34.887023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:34.887031 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:31:34.887111 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:34.887112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:34.887138 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:34.887211 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:31:34.887352 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.887837 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.888603 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:34.888799 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:34.888874 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:34.889047 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:31:34.889147 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:31:34.889238 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:34.889258 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 15:31:34.889278 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:31:34.890904 25728 raft_consensus.cc:1261] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 1 FOLLOWER]: Refusing update from remote peer e01b9f5f2178411ab9a117a65b635360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:34.890908 25462 raft_consensus.cc:1261] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 1 FOLLOWER]: Refusing update from remote peer e01b9f5f2178411ab9a117a65b635360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:31:34.891266 25797 consensus_queue.cc:1035] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9addd93b3c2d4f879838cc0a54df4bb2" member_type: VOTER last_known_addr { host: "127.5.98.195" port: 45777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:34.891391 25799 consensus_queue.cc:1035] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381872cd1a4748339459c8b5ba883647" member_type: VOTER last_known_addr { host: "127.5.98.193" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:31:34.893411 25802 mvcc.cc:205] Tried to move back new op lower bound from 7332920913470992384 to 7332920913388318720. Current Snapshot: MvccSnapshot[applied={T|T < 7332920913470992384}]
I20260924 15:31:34.895987 25320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:34.896083 25320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:31:34.896828 25320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 743222586333499e8a0397c2e001f9ec: Sending DeleteTablet for 3 replicas of tablet 151e46d91b684057abba75ccbdd5ba60
I20260924 15:31:34.897207 25442 tablet_service.cc:1514] Processing DeleteTablet for tablet 151e46d91b684057abba75ccbdd5ba60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:34 UTC) from {username='slave'} at 127.0.0.1:52172
I20260924 15:31:34.897348 25708 tablet_service.cc:1514] Processing DeleteTablet for tablet 151e46d91b684057abba75ccbdd5ba60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:34 UTC) from {username='slave'} at 127.0.0.1:39114
I20260924 15:31:34.897383 25576 tablet_service.cc:1514] Processing DeleteTablet for tablet 151e46d91b684057abba75ccbdd5ba60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:31:34 UTC) from {username='slave'} at 127.0.0.1:39260
I20260924 15:31:34.897540 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25380
I20260924 15:31:34.897589 25811 tablet_replica.cc:330] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647: stopping tablet replica
I20260924 15:31:34.897689 25811 raft_consensus.cc:2229] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:34.897778 25811 raft_consensus.cc:2258] T 151e46d91b684057abba75ccbdd5ba60 P 381872cd1a4748339459c8b5ba883647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:34.899782 25813 tablet_replica.cc:330] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: stopping tablet replica
I20260924 15:31:34.899873 25813 raft_consensus.cc:2229] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:31:34.899955 25813 raft_consensus.cc:2258] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:34.900097 25812 tablet_replica.cc:330] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: stopping tablet replica
I20260924 15:31:34.900178 25812 raft_consensus.cc:2229] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:31:34.900267 25812 raft_consensus.cc:2258] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:31:34.900704 25812 ts_tablet_manager.cc:1905] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:34.900799 25813 ts_tablet_manager.cc:1905] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:31:34.902495 25813 ts_tablet_manager.cc:1918] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:34.902572 25812 ts_tablet_manager.cc:1918] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:31:34.902648 25812 log.cc:1196] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-2/wal/wals/151e46d91b684057abba75ccbdd5ba60
I20260924 15:31:34.902665 25813 log.cc:1196] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Deleting WAL directory at /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263655576751-5515-0/minicluster-data/ts-1/wal/wals/151e46d91b684057abba75ccbdd5ba60
I20260924 15:31:34.902940 25812 ts_tablet_manager.cc:1939] T 151e46d91b684057abba75ccbdd5ba60 P 9addd93b3c2d4f879838cc0a54df4bb2: Deleting consensus metadata
I20260924 15:31:34.903121 25813 ts_tablet_manager.cc:1939] T 151e46d91b684057abba75ccbdd5ba60 P e01b9f5f2178411ab9a117a65b635360: Deleting consensus metadata
I20260924 15:31:34.903342 25307 catalog_manager.cc:4841] TS 9addd93b3c2d4f879838cc0a54df4bb2 (127.5.98.195:45777): tablet 151e46d91b684057abba75ccbdd5ba60 (table test-table [id=33cfe77ee1994413a9ec5d6cd17e6ac3]) successfully deleted
W20260924 15:31:34.903642 25306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.98.193:39607 (error 108)
I20260924 15:31:34.903937 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25512
I20260924 15:31:34.903955 25307 catalog_manager.cc:4841] TS e01b9f5f2178411ab9a117a65b635360 (127.5.98.194:38471): tablet 151e46d91b684057abba75ccbdd5ba60 (table test-table [id=33cfe77ee1994413a9ec5d6cd17e6ac3]) successfully deleted
I20260924 15:31:34.904500 25787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:34.904030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.193:39607 (local address 127.0.0.1:52174)
0924 15:31:34.904430 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:34.904469 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.193:39607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260924 15:31:34.904708 25306 catalog_manager.cc:4584] TS 381872cd1a4748339459c8b5ba883647 (127.5.98.193:39607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 151e46d91b684057abba75ccbdd5ba60: Network error: Client connection negotiation failed: client connection to 127.5.98.193:39607: connect: Connection refused (error 111)
I20260924 15:31:34.909142 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25646
I20260924 15:31:34.913831 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25288
2026-09-24T15:31:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (729 ms)
[----------] 6 tests from EncryptionPolicyTest (8657 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:34 dist-test-slave-4g6f krb5kdc[25817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:34 dist-test-slave-4g6f krb5kdc[25817](info): set up 2 sockets
Sep 24 15:31:34 dist-test-slave-4g6f krb5kdc[25817](info): commencing operation
krb5kdc: starting...
W20260924 15:31:35.980599 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:36.094517 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:36.172581 25830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:36.172775 25830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:36.172797 25830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:36.173854 25830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:36.173887 25830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:36.173902 25830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:36.173914 25830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:36.173926 25830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:36.175303 25830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:36.175485 25830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:36.175643 25830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:36.180341 25838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:36.180335 25836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:36.180426 25835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:36.180723 25830 server_base.cc:1034] running on GCE node
I20260924 15:31:36.180945 25830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:36.181182 25830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:36.182319 25830 hybrid_clock.cc:648] HybridClock initialized: now 1790263896182306 us; error 28 us; skew 500 ppm
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:36.184778 25830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:36.185249 25830 webserver.cc:460] Webserver started at http://127.0.0.1:44243/ using document root <none> and password file <none>
I20260924 15:31:36.185469 25830 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:36.185537 25830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:36.185688 25830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:36.186697 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "e09609acf6ab479f90aededb05824e75"
format_stamp: "Formatted at 2026-09-24 15:31:36 on dist-test-slave-4g6f"
server_key: "4fc89ea3e76646a535c6948756604f87"
server_key_iv: "0288b8d413f224d5b81e3fada1fdc1ca"
server_key_version: "encryptionkey@0"
I20260924 15:31:36.187057 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "e09609acf6ab479f90aededb05824e75"
format_stamp: "Formatted at 2026-09-24 15:31:36 on dist-test-slave-4g6f"
server_key: "4fc89ea3e76646a535c6948756604f87"
server_key_iv: "0288b8d413f224d5b81e3fada1fdc1ca"
server_key_version: "encryptionkey@0"
I20260924 15:31:36.188441 25830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:36.189158 25845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:36.189316 25830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:36.189420 25830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "e09609acf6ab479f90aededb05824e75"
format_stamp: "Formatted at 2026-09-24 15:31:36 on dist-test-slave-4g6f"
server_key: "4fc89ea3e76646a535c6948756604f87"
server_key_iv: "0288b8d413f224d5b81e3fada1fdc1ca"
server_key_version: "encryptionkey@0"
I20260924 15:31:36.189494 25830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:36.231280 25830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:36.233409 25830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:36.233592 25830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:36.237685 25830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37625
I20260924 15:31:36.237721 25897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37625 every 8 connection(s)
I20260924 15:31:36.238078 25830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:36.238893 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25830
I20260924 15:31:36.239010 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:36.239218 5515 external_mini_cluster.cc:1428] Setting key 65e2b489cd4c6c8f1fecbead7c4a65ad
I20260924 15:31:36.239214 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:36.242238 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75: Bootstrap starting.
I20260924 15:31:36.242893 25898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:36.243172 25898 log.cc:824] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:36.243968 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75: No bootstrap required, opened a new log
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:31:36.245965 25898 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } }
I20260924 15:31:36.246121 25898 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:36.246166 25898 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09609acf6ab479f90aededb05824e75, State: Initialized, Role: FOLLOWER
I20260924 15:31:36.246325 25898 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } }
I20260924 15:31:36.246435 25898 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:36.246474 25898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:36.246528 25898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:36.247265 25898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } }
I20260924 15:31:36.247390 25898 leader_election.cc:302] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09609acf6ab479f90aededb05824e75; no voters:
I20260924 15:31:36.247555 25898 leader_election.cc:288] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:36.247599 25903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:36.247757 25903 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [term 1 LEADER]: Becoming Leader. State: Replica: e09609acf6ab479f90aededb05824e75, State: Running, Role: LEADER
I20260924 15:31:36.247810 25898 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:36.247882 25903 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } }
I20260924 15:31:36.248283 25903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09609acf6ab479f90aededb05824e75. Latest consensus state: current_term: 1 leader_uuid: "e09609acf6ab479f90aededb05824e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } } }
I20260924 15:31:36.248437 25903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:36.248680 25911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:36.248288 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09609acf6ab479f90aededb05824e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09609acf6ab479f90aededb05824e75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37625 } } }
I20260924 15:31:36.248790 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:36.249307 25911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:36.250273 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:36.239790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32875 (local address 127.0.0.1:37625)
0924 15:31:36.239988 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:36.239997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:36.240035 (+ 38us) server_negotiation.cc:407] Connection header received
0924 15:31:36.240647 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:36.240654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:36.240700 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:36.240854 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:36.241526 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:36.242453 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:36.243317 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:36.243908 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:36.244024 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:36.246330 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:36.246334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:36.246335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:36.246346 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:36.248335 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:36.248790 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:36.248793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:36.248794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:36.248942 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:36.249221 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:36.249222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:36.249223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:36.249550 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:36.249719 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:36.250010 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 15:31:36.250121 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260924 15:31:36.251346 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:36.252899 25911 catalog_manager.cc:1347] Generated new cluster ID: 3969d0e89a4346f1b5126e8f341c940c
I20260924 15:31:36.252964 25911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:36.262586 25901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:36.252902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36133 (local address 127.0.0.1:37625)
0924 15:31:36.255772 (+ 2870us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:36.255773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:36.255782 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:31:36.255802 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:36.255803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:36.255811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:36.255882 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:36.256502 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:36.257177 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:36.258167 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:36.258481 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:36.258536 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:36.259884 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:36.259885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:36.259885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:36.259892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:36.261372 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:36.261791 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:36.261792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:36.261793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:36.261867 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:36.262098 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:36.262098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:36.262099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:36.262206 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:36.262297 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:36.262450 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:31:36.262506 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2850}
I20260924 15:31:36.272293 25911 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:36.272825 25911 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:36.279237 25911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09609acf6ab479f90aededb05824e75: Generated new TSK 0
I20260924 15:31:36.279423 25911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:36.284121 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25830
W20260924 15:31:36.288980 25815 connection.cc:535] client connection to 127.0.0.1:37625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37625): Connection reset by peer (error 104)
2026-09-24T15:31:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25930](info): set up 2 sockets
Sep 24 15:31:36 dist-test-slave-4g6f krb5kdc[25930](info): commencing operation
krb5kdc: starting...
W20260924 15:31:37.320503 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:37 dist-test-slave-4g6f krb5kdc[25930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:37.433832 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:37.512908 25943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:37.513093 25943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:37.513118 25943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:37.514199 25943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:37.514238 25943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:37.514252 25943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:37.514286 25943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:37.514317 25943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:37.515820 25943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:37.516065 25943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:37.516292 25943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:37.521070 25951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:37.521052 25948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:37.521056 25949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:37.521085 25943 server_base.cc:1034] running on GCE node
I20260924 15:31:37.521588 25943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:37.521939 25943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:37.523118 25943 hybrid_clock.cc:648] HybridClock initialized: now 1790263897523102 us; error 27 us; skew 500 ppm
Sep 24 15:31:37 dist-test-slave-4g6f krb5kdc[25930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:37.525411 25943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:37.525811 25943 webserver.cc:460] Webserver started at http://127.0.0.1:42259/ using document root <none> and password file <none>
I20260924 15:31:37.526055 25943 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:37.526125 25943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:37.526270 25943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:37.527246 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "5f296941011844e993e123bf641c870b"
format_stamp: "Formatted at 2026-09-24 15:31:37 on dist-test-slave-4g6f"
server_key: "31848de6ff8dc92b10fefc0021050385"
server_key_iv: "36e4c9a0b5ff4ff104c7fd81523d606a"
server_key_version: "encryptionkey@0"
I20260924 15:31:37.527604 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "5f296941011844e993e123bf641c870b"
format_stamp: "Formatted at 2026-09-24 15:31:37 on dist-test-slave-4g6f"
server_key: "31848de6ff8dc92b10fefc0021050385"
server_key_iv: "36e4c9a0b5ff4ff104c7fd81523d606a"
server_key_version: "encryptionkey@0"
I20260924 15:31:37.528865 25943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:37.529683 25958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:37.529883 25943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:37.529986 25943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "5f296941011844e993e123bf641c870b"
format_stamp: "Formatted at 2026-09-24 15:31:37 on dist-test-slave-4g6f"
server_key: "31848de6ff8dc92b10fefc0021050385"
server_key_iv: "36e4c9a0b5ff4ff104c7fd81523d606a"
server_key_version: "encryptionkey@0"
I20260924 15:31:37.530071 25943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:37.539999 25943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:37.541685 25943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:37.541842 25943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:37.545863 25943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39961
I20260924 15:31:37.545900 26010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39961 every 8 connection(s)
I20260924 15:31:37.546247 25943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:37.547288 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:37.548048 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 25943
I20260924 15:31:37.548193 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:37.548372 5515 external_mini_cluster.cc:1428] Setting key 1baea7ccd5a7e3013ad4d62a0b2f29af
I20260924 15:31:37.550530 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b: Bootstrap starting.
I20260924 15:31:37.551199 26011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:37.551512 26011 log.cc:824] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:37.552209 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b: No bootstrap required, opened a new log
I20260924 15:31:37.554270 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:37.548899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52129 (local address 127.0.0.1:39961)
0924 15:31:37.549094 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:37.549103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:37.549135 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:31:37.549669 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:37.549678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:37.549740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:37.549922 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:37.550089 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:37.551458 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:37.552398 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:37.552802 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:37.552952 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:37.553649 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:37.553656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:37.553656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:37.553815 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:37.553842 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:37.553852 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:37.553946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:37.554081 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 15:31:37.554110 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260924 15:31:37.554203 26011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260924 15:31:37.554389 26011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:37.554443 26011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f296941011844e993e123bf641c870b, State: Initialized, Role: FOLLOWER
I20260924 15:31:37.554559 26011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260924 15:31:37.554644 26011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:37.554678 26011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:37.554723 26011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:37.555260 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:37.555469 26011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260924 15:31:37.555583 26011 leader_election.cc:302] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f296941011844e993e123bf641c870b; no voters:
I20260924 15:31:37.555763 26011 leader_election.cc:288] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:37.555792 26016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:37.555960 26016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [term 1 LEADER]: Becoming Leader. State: Replica: 5f296941011844e993e123bf641c870b, State: Running, Role: LEADER
I20260924 15:31:37.556022 26011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:37.556160 26016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } }
I20260924 15:31:37.556561 26016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f296941011844e993e123bf641c870b. Latest consensus state: current_term: 1 leader_uuid: "5f296941011844e993e123bf641c870b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } } }
I20260924 15:31:37.556572 26018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f296941011844e993e123bf641c870b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f296941011844e993e123bf641c870b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39961 } } }
I20260924 15:31:37.556739 26018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:37.556950 26016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:37.557135 26031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:37.557622 26031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:37.559188 26031 catalog_manager.cc:1347] Generated new cluster ID: f7c7a819e1494957ab05b336bbf3e41b
I20260924 15:31:37.559238 26031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:37.560355 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:37.556746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58711 (local address 127.0.0.1:39961)
0924 15:31:37.556781 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:37.556783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:37.556980 (+ 197us) server_negotiation.cc:407] Connection header received
0924 15:31:37.557056 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:37.557057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:37.557066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:37.557115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:37.557301 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:37.558044 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:37.559057 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:37.559364 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:37.559437 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:37.560100 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:37.560102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:37.560102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:37.560106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:37.560120 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:37.560124 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:37.560180 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:37.560269 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:31:37.560279 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:31:37.567072 26031 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:37.567610 26031 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:37.571707 26031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f296941011844e993e123bf641c870b: Generated new TSK 0
I20260924 15:31:37.571884 26031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:37.580075 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 25943
W20260924 15:31:37.584842 25928 connection.cc:535] client connection to 127.0.0.1:39961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39961): Connection reset by peer (error 104)
2026-09-24T15:31:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:37 dist-test-slave-4g6f krb5kdc[26044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:37 dist-test-slave-4g6f krb5kdc[26044](info): set up 2 sockets
Sep 24 15:31:37 dist-test-slave-4g6f krb5kdc[26044](info): commencing operation
krb5kdc: starting...
W20260924 15:31:38.616861 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:38 dist-test-slave-4g6f krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:38.730357 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:38.809825 26057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:38.810115 26057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:38.810153 26057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:38.811266 26057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:38.811332 26057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:38.811357 26057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:38.811385 26057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:38.811411 26057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:38.812933 26057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:38.813197 26057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:38.813442 26057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:38.818219 26065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:38.818234 26062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:38.818233 26063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:38.818298 26057 server_base.cc:1034] running on GCE node
I20260924 15:31:38.818678 26057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:38.818939 26057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:38.820107 26057 hybrid_clock.cc:648] HybridClock initialized: now 1790263898820099 us; error 49 us; skew 500 ppm
Sep 24 15:31:38 dist-test-slave-4g6f krb5kdc[26044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:38.827905 26057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:38.828321 26057 webserver.cc:460] Webserver started at http://127.0.0.1:37727/ using document root <none> and password file <none>
I20260924 15:31:38.828563 26057 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:38.828632 26057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:38.828784 26057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:38.829782 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "0ed1a1ffa0264a93821532c05e2f3a48"
format_stamp: "Formatted at 2026-09-24 15:31:38 on dist-test-slave-4g6f"
server_key: "82e7de78faf1a8ce4e014cc13c86772b"
server_key_iv: "201a626dc0393f9b5c157f7235b7ca69"
server_key_version: "encryptionkey@0"
I20260924 15:31:38.830128 26057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "0ed1a1ffa0264a93821532c05e2f3a48"
format_stamp: "Formatted at 2026-09-24 15:31:38 on dist-test-slave-4g6f"
server_key: "82e7de78faf1a8ce4e014cc13c86772b"
server_key_iv: "201a626dc0393f9b5c157f7235b7ca69"
server_key_version: "encryptionkey@0"
I20260924 15:31:38.831382 26057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:38.832248 26072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:38.832439 26057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:31:38.832553 26057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "0ed1a1ffa0264a93821532c05e2f3a48"
format_stamp: "Formatted at 2026-09-24 15:31:38 on dist-test-slave-4g6f"
server_key: "82e7de78faf1a8ce4e014cc13c86772b"
server_key_iv: "201a626dc0393f9b5c157f7235b7ca69"
server_key_version: "encryptionkey@0"
I20260924 15:31:38.832640 26057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:38.835894 26057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:38.837586 26057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:38.837739 26057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:38.841722 26057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46513
I20260924 15:31:38.841750 26125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46513 every 8 connection(s)
I20260924 15:31:38.842052 26057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:38.842986 26126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:38.844511 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26057
I20260924 15:31:38.844610 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:38.844813 5515 external_mini_cluster.cc:1428] Setting key a8cdf452d0db82e4642b66eb16ac5d01
I20260924 15:31:38.845544 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48: Bootstrap starting.
I20260924 15:31:38.846203 26126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:38.846549 26126 log.cc:824] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:38.847330 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48: No bootstrap required, opened a new log
I20260924 15:31:38.847440 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:38.845321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44235 (local address 127.0.0.1:46513)
0924 15:31:38.845527 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:38.845536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:38.845567 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:31:38.846125 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:38.846134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:38.846168 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:38.846321 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:38.847008 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:38.847012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:38.847013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:38.847018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:38.847064 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:38.847079 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:38.847178 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:38.847208 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 15:31:38.847244 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260924 15:31:38.848510 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:38.850008 26126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260924 15:31:38.850255 26126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:38.850327 26126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed1a1ffa0264a93821532c05e2f3a48, State: Initialized, Role: FOLLOWER
I20260924 15:31:38.850513 26126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260924 15:31:38.850626 26126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:38.850660 26126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:38.850719 26126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:38.851210 26126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260924 15:31:38.851279 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:38.850185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59403 (local address 127.0.0.1:46513)
0924 15:31:38.850225 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:38.850226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:38.850311 (+ 85us) server_negotiation.cc:407] Connection header received
0924 15:31:38.850391 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:38.850392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:38.850396 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:38.850459 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:38.851028 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:38.851029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:38.851029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:38.851033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:38.851046 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:38.851049 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:38.851092 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:38.851144 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 15:31:38.851158 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:31:38.851354 26126 leader_election.cc:302] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ed1a1ffa0264a93821532c05e2f3a48; no voters:
I20260924 15:31:38.851511 26126 leader_election.cc:288] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:38.851543 26137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:38.851691 26137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed1a1ffa0264a93821532c05e2f3a48, State: Running, Role: LEADER
I20260924 15:31:38.851778 26126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:38.851841 26137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } }
I20260924 15:31:38.852197 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed1a1ffa0264a93821532c05e2f3a48. Latest consensus state: current_term: 1 leader_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } } }
I20260924 15:31:38.852241 26138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed1a1ffa0264a93821532c05e2f3a48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46513 } } }
I20260924 15:31:38.852326 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:38.852396 26138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:38.852800 26146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:38.853242 26146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:38.854691 26146 catalog_manager.cc:1347] Generated new cluster ID: 6cfd685ab8534fc7ab2ccf47b64ded1b
I20260924 15:31:38.854760 26146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:38.882812 26146 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:38.882998 26146 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:38.885924 26146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ed1a1ffa0264a93821532c05e2f3a48: Generated new TSK 0
I20260924 15:31:38.886119 26146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:38.893671 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26057
2026-09-24T15:31:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:38 dist-test-slave-4g6f krb5kdc[26158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:38 dist-test-slave-4g6f krb5kdc[26158](info): set up 2 sockets
Sep 24 15:31:38 dist-test-slave-4g6f krb5kdc[26158](info): commencing operation
krb5kdc: starting...
W20260924 15:31:39.928753 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:40.041776 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:40.121567 26171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:40.121806 26171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:40.121843 26171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:40.122985 26171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:40.123051 26171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:40.123075 26171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:40.123096 26171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:40.123126 26171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:40.124617 26171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:40.124891 26171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:40.125114 26171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:40.130046 26179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:40.130151 26171 server_base.cc:1034] running on GCE node
W20260924 15:31:40.130064 26177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:40.130049 26176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:40.130528 26171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:40.130818 26171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:40.131996 26171 hybrid_clock.cc:648] HybridClock initialized: now 1790263900131968 us; error 44 us; skew 500 ppm
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:40.134397 26171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:40.134862 26171 webserver.cc:460] Webserver started at http://127.0.0.1:44769/ using document root <none> and password file <none>
I20260924 15:31:40.135100 26171 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:40.135169 26171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:40.135303 26171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:40.136430 26171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "bee63ebede4c4eb79c06c28d616d5daf"
format_stamp: "Formatted at 2026-09-24 15:31:40 on dist-test-slave-4g6f"
server_key: "e96eba525de4ec51dd2a7b6f66f1f7fd"
server_key_iv: "16298f83509e4d7aa6403674d47cc9cc"
server_key_version: "encryptionkey@0"
I20260924 15:31:40.136829 26171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "bee63ebede4c4eb79c06c28d616d5daf"
format_stamp: "Formatted at 2026-09-24 15:31:40 on dist-test-slave-4g6f"
server_key: "e96eba525de4ec51dd2a7b6f66f1f7fd"
server_key_iv: "16298f83509e4d7aa6403674d47cc9cc"
server_key_version: "encryptionkey@0"
I20260924 15:31:40.138172 26171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:31:40.138958 26186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:40.139168 26171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:40.139276 26171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "bee63ebede4c4eb79c06c28d616d5daf"
format_stamp: "Formatted at 2026-09-24 15:31:40 on dist-test-slave-4g6f"
server_key: "e96eba525de4ec51dd2a7b6f66f1f7fd"
server_key_iv: "16298f83509e4d7aa6403674d47cc9cc"
server_key_version: "encryptionkey@0"
I20260924 15:31:40.139360 26171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:40.152299 26171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:40.154045 26171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:40.154206 26171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:40.158532 26171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44055
I20260924 15:31:40.158581 26239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44055 every 8 connection(s)
I20260924 15:31:40.158937 26171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:40.159924 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:40.162231 26240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf: Bootstrap starting.
I20260924 15:31:40.162818 26240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:40.163111 26240 log.cc:824] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:40.163865 26240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf: No bootstrap required, opened a new log
I20260924 15:31:40.165445 26240 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260924 15:31:40.165583 26240 raft_consensus.cc:374] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:40.165606 26240 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee63ebede4c4eb79c06c28d616d5daf, State: Initialized, Role: FOLLOWER
I20260924 15:31:40.165757 26240 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260924 15:31:40.165822 26240 raft_consensus.cc:388] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:40.165887 26240 raft_consensus.cc:482] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:40.165948 26240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:40.166038 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26171
I20260924 15:31:40.166148 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:40.166349 5515 external_mini_cluster.cc:1428] Setting key c344907877cec67bf70051454cdbddd7
I20260924 15:31:40.166630 26240 raft_consensus.cc:504] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260924 15:31:40.166749 26240 leader_election.cc:302] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bee63ebede4c4eb79c06c28d616d5daf; no voters:
I20260924 15:31:40.166930 26240 leader_election.cc:288] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:40.167099 26246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:40.167182 26240 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:40.167253 26246 raft_consensus.cc:686] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [term 1 LEADER]: Becoming Leader. State: Replica: bee63ebede4c4eb79c06c28d616d5daf, State: Running, Role: LEADER
I20260924 15:31:40.167382 26246 consensus_queue.cc:237] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260924 15:31:40.167706 26246 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bee63ebede4c4eb79c06c28d616d5daf. Latest consensus state: current_term: 1 leader_uuid: "bee63ebede4c4eb79c06c28d616d5daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260924 15:31:40.167707 26247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bee63ebede4c4eb79c06c28d616d5daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bee63ebede4c4eb79c06c28d616d5daf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260924 15:31:40.167867 26247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:40.168067 26246 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:40.168495 26250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:40.169064 26250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:40.170785 26250 catalog_manager.cc:1347] Generated new cluster ID: 16b1c92e425e4ae584b6d96e241e24b9
I20260924 15:31:40.170846 26250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:31:40.178306 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:40.166945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45941 (local address 127.0.0.1:44055)
0924 15:31:40.167118 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:40.167127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:40.167157 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:31:40.167298 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:40.167306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:40.167381 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:40.168501 (+ 1120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:40.168627 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:40.169948 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:40.170995 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:40.171580 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:40.171703 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:40.174467 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:40.174471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:40.174472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:40.174483 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:40.176571 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:40.176965 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:40.176968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:40.176969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:40.177089 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:40.177294 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:40.177295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:40.177296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:40.177604 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:40.177770 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:40.177837 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:31:40.178094 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 15:31:40.179339 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:40.188030 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:40.180875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54627 (local address 127.0.0.1:44055)
0924 15:31:40.180924 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:40.180926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:40.181046 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:31:40.181139 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:40.181140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:40.181149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:40.181177 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:40.181846 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:40.182579 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:40.183610 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:40.183973 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:40.184014 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:40.185359 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:40.185360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:40.185361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:40.185367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:40.186815 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:40.187205 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:40.187206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:40.187206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:40.187257 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:40.187445 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:40.187446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:40.187446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:40.187562 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:40.187631 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:40.187855 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 15:31:40.187942 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:31:40.212023 26250 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:40.212780 26250 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:40.217864 26250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bee63ebede4c4eb79c06c28d616d5daf: Generated new TSK 0
I20260924 15:31:40.218053 26250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:40.222014 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26171
2026-09-24T15:31:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26273](info): set up 2 sockets
Sep 24 15:31:40 dist-test-slave-4g6f krb5kdc[26273](info): commencing operation
krb5kdc: starting...
W20260924 15:31:41.260668 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:41 dist-test-slave-4g6f krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:41.374143 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:41.453573 26286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:41.453769 26286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:41.453792 26286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:41.454952 26286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:41.455000 26286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:41.455014 26286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:41.455027 26286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:41.455055 26286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:41.456574 26286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:41.456828 26286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:41.457068 26286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:41.462023 26291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:41.462035 26292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:41.462065 26294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:41.462291 26286 server_base.cc:1034] running on GCE node
I20260924 15:31:41.462581 26286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:41.462881 26286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:41.464056 26286 hybrid_clock.cc:648] HybridClock initialized: now 1790263901464049 us; error 43 us; skew 500 ppm
Sep 24 15:31:41 dist-test-slave-4g6f krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:41.466591 26286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:41.467048 26286 webserver.cc:460] Webserver started at http://127.0.0.1:34891/ using document root <none> and password file <none>
I20260924 15:31:41.467249 26286 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:41.467289 26286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:41.467381 26286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:41.468438 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "22092756ff134f5eaa8b04471dadf24e"
format_stamp: "Formatted at 2026-09-24 15:31:41 on dist-test-slave-4g6f"
server_key: "66db9919b8b53fabf86b4d9109e9df9d"
server_key_iv: "197309c49b837425c5114895de0f2206"
server_key_version: "encryptionkey@0"
I20260924 15:31:41.468742 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "22092756ff134f5eaa8b04471dadf24e"
format_stamp: "Formatted at 2026-09-24 15:31:41 on dist-test-slave-4g6f"
server_key: "66db9919b8b53fabf86b4d9109e9df9d"
server_key_iv: "197309c49b837425c5114895de0f2206"
server_key_version: "encryptionkey@0"
I20260924 15:31:41.469995 26286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:31:41.470732 26301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:41.470952 26286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:41.471035 26286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "22092756ff134f5eaa8b04471dadf24e"
format_stamp: "Formatted at 2026-09-24 15:31:41 on dist-test-slave-4g6f"
server_key: "66db9919b8b53fabf86b4d9109e9df9d"
server_key_iv: "197309c49b837425c5114895de0f2206"
server_key_version: "encryptionkey@0"
I20260924 15:31:41.471091 26286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:41.484582 26286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:41.486557 26286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:41.486670 26286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:41.490589 26286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38069
I20260924 15:31:41.490674 26354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38069 every 8 connection(s)
I20260924 15:31:41.490986 26286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:41.492061 26355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:41.494277 26355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e: Bootstrap starting.
I20260924 15:31:41.494853 26355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:41.495152 26355 log.cc:824] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:41.495898 26355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e: No bootstrap required, opened a new log
I20260924 15:31:41.497591 26355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260924 15:31:41.497753 26355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:41.497802 26355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22092756ff134f5eaa8b04471dadf24e, State: Initialized, Role: FOLLOWER
I20260924 15:31:41.497938 26355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260924 15:31:41.498024 26355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:41.498065 26355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:41.498121 26355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:41.498507 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26286
I20260924 15:31:41.498621 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:41.498842 5515 external_mini_cluster.cc:1428] Setting key 4cf1b333929f1581d24167bb23c3f5b7
I20260924 15:31:41.498863 26355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260924 15:31:41.498965 26355 leader_election.cc:302] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22092756ff134f5eaa8b04471dadf24e; no voters:
I20260924 15:31:41.499132 26355 leader_election.cc:288] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:41.499156 26358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:41.499270 26358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [term 1 LEADER]: Becoming Leader. State: Replica: 22092756ff134f5eaa8b04471dadf24e, State: Running, Role: LEADER
I20260924 15:31:41.499395 26355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:41.499408 26358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } }
I20260924 15:31:41.500139 26359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22092756ff134f5eaa8b04471dadf24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } } }
I20260924 15:31:41.500164 26360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22092756ff134f5eaa8b04471dadf24e. Latest consensus state: current_term: 1 leader_uuid: "22092756ff134f5eaa8b04471dadf24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22092756ff134f5eaa8b04471dadf24e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38069 } } }
I20260924 15:31:41.500267 26359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:41.500300 26360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:41.500687 26370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:41.501323 26370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:41.502871 26370 catalog_manager.cc:1347] Generated new cluster ID: b30bec80c9b44d9cabfc1f7151d96e02
I20260924 15:31:41.502923 26370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:41.506047 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:41.499532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46187 (local address 127.0.0.1:38069)
0924 15:31:41.499719 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:41.499729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:41.500701 (+ 972us) server_negotiation.cc:407] Connection header received
0924 15:31:41.500827 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:41.500834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:41.500915 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:41.501049 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:41.501403 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:41.502802 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:41.503976 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:41.504586 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:41.504683 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:41.505380 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:41.505384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:41.505384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:41.505578 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:41.505614 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:41.505627 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:41.505687 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:41.505785 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:31:41.505847 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260924 15:31:41.507148 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:41.511909 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:41.508546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60301 (local address 127.0.0.1:38069)
0924 15:31:41.508577 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:41.508579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:41.508683 (+ 104us) server_negotiation.cc:407] Connection header received
0924 15:31:41.508737 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:41.508738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:41.508747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:41.508801 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:41.508931 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:41.509637 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:41.510638 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:41.510919 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:41.510979 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:41.511643 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:41.511644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:41.511644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:41.511648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:41.511682 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:41.511686 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:41.511739 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:41.511821 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:31:41.511843 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:31:41.528330 26370 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:41.529038 26370 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:41.533201 26370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22092756ff134f5eaa8b04471dadf24e: Generated new TSK 0
I20260924 15:31:41.533375 26370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:41.536235 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26286
2026-09-24T15:31:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:41 dist-test-slave-4g6f krb5kdc[26387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:41 dist-test-slave-4g6f krb5kdc[26387](info): set up 2 sockets
Sep 24 15:31:41 dist-test-slave-4g6f krb5kdc[26387](info): commencing operation
krb5kdc: starting...
W20260924 15:31:42.592903 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:42 dist-test-slave-4g6f krb5kdc[26387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:42.708230 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:42.787739 26400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:42.787976 26400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:42.788017 26400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:42.789175 26400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:42.789242 26400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:42.789266 26400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:42.789292 26400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:42.789323 26400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:42.790786 26400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:42.791077 26400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:42.791312 26400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:42.796133 26406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:42.796113 26409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:42.796335 26400 server_base.cc:1034] running on GCE node
W20260924 15:31:42.796386 26407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:42.796614 26400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:42.796912 26400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:42.798096 26400 hybrid_clock.cc:648] HybridClock initialized: now 1790263902798078 us; error 41 us; skew 500 ppm
Sep 24 15:31:42 dist-test-slave-4g6f krb5kdc[26387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:42.800695 26400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:42.801168 26400 webserver.cc:460] Webserver started at http://127.0.0.1:38227/ using document root <none> and password file <none>
I20260924 15:31:42.801364 26400 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:42.801409 26400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:42.801503 26400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:42.802574 26400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "0d3616ffa8dd4439961757bec90b73c5"
format_stamp: "Formatted at 2026-09-24 15:31:42 on dist-test-slave-4g6f"
server_key: "9d403a71c7b712413a313341e9a932e1"
server_key_iv: "a24a210cf10be54aef85f5b6157745d2"
server_key_version: "encryptionkey@0"
I20260924 15:31:42.802878 26400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "0d3616ffa8dd4439961757bec90b73c5"
format_stamp: "Formatted at 2026-09-24 15:31:42 on dist-test-slave-4g6f"
server_key: "9d403a71c7b712413a313341e9a932e1"
server_key_iv: "a24a210cf10be54aef85f5b6157745d2"
server_key_version: "encryptionkey@0"
I20260924 15:31:42.804279 26400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:42.805053 26416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:42.805258 26400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:42.805356 26400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "0d3616ffa8dd4439961757bec90b73c5"
format_stamp: "Formatted at 2026-09-24 15:31:42 on dist-test-slave-4g6f"
server_key: "9d403a71c7b712413a313341e9a932e1"
server_key_iv: "a24a210cf10be54aef85f5b6157745d2"
server_key_version: "encryptionkey@0"
I20260924 15:31:42.805433 26400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:42.808887 26400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:42.810871 26400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:42.811033 26400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:42.815227 26400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43883
I20260924 15:31:42.815256 26468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43883 every 8 connection(s)
I20260924 15:31:42.815598 26400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:42.816566 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:42.818819 26469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5: Bootstrap starting.
I20260924 15:31:42.819380 26469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:42.819712 26469 log.cc:824] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:42.820381 26469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5: No bootstrap required, opened a new log
I20260924 15:31:42.821992 26469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } }
I20260924 15:31:42.822132 26469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:42.822155 26469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3616ffa8dd4439961757bec90b73c5, State: Initialized, Role: FOLLOWER
I20260924 15:31:42.822304 26469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } }
I20260924 15:31:42.822368 26469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:42.822389 26469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:42.822451 26469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:42.822580 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26400
I20260924 15:31:42.822724 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:42.822907 5515 external_mini_cluster.cc:1428] Setting key b76a105bed9d386b101b196bc38318cb
I20260924 15:31:42.823168 26469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } }
I20260924 15:31:42.823338 26469 leader_election.cc:302] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d3616ffa8dd4439961757bec90b73c5; no voters:
I20260924 15:31:42.823578 26469 leader_election.cc:288] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:42.823764 26474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:42.823928 26469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:42.823954 26474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3616ffa8dd4439961757bec90b73c5, State: Running, Role: LEADER
I20260924 15:31:42.824168 26474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } }
I20260924 15:31:42.824533 26475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d3616ffa8dd4439961757bec90b73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } } }
I20260924 15:31:42.824658 26475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:42.824896 26476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3616ffa8dd4439961757bec90b73c5. Latest consensus state: current_term: 1 leader_uuid: "0d3616ffa8dd4439961757bec90b73c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d3616ffa8dd4439961757bec90b73c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43883 } } }
I20260924 15:31:42.825003 26476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:42.825314 26482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:42.825852 26482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:42.827471 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:42.823418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55623 (local address 127.0.0.1:43883)
0924 15:31:42.823692 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:42.823703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:42.824540 (+ 837us) server_negotiation.cc:407] Connection header received
0924 15:31:42.824769 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:42.824778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:42.824811 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:42.824926 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:42.826893 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:42.826901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:42.826902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:42.826926 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:42.826986 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:42.826998 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:42.827074 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:42.827177 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 15:31:42.827255 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260924 15:31:42.827950 26482 catalog_manager.cc:1347] Generated new cluster ID: f6219884021e43a0ae8beceaa390786f
I20260924 15:31:42.828006 26482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:42.828675 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:42.832813 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:42.830035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35313 (local address 127.0.0.1:43883)
0924 15:31:42.830070 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:42.830072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:42.831879 (+ 1807us) server_negotiation.cc:407] Connection header received
0924 15:31:42.831958 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:42.831959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:42.831966 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:42.832011 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:42.832583 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:42.832584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:42.832584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:42.832589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:42.832604 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:42.832608 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:42.832666 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:42.832722 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 15:31:42.832737 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:42.854658 26482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:42.854879 26482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:42.865236 26482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d3616ffa8dd4439961757bec90b73c5: Generated new TSK 0
I20260924 15:31:42.865448 26482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:42.869705 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26400
2026-09-24T15:31:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:42 dist-test-slave-4g6f krb5kdc[26502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:42 dist-test-slave-4g6f krb5kdc[26502](info): set up 2 sockets
Sep 24 15:31:42 dist-test-slave-4g6f krb5kdc[26502](info): commencing operation
krb5kdc: starting...
W20260924 15:31:43.928745 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:44.048345 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:44.131313 26515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:44.131552 26515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:44.131588 26515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:44.132763 26515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:44.132838 26515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:44.132866 26515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:44.132897 26515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:44.132920 26515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:44.134392 26515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:44.134665 26515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:44.134891 26515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:44.139895 26523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:44.139882 26520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:44.139899 26515 server_base.cc:1034] running on GCE node
W20260924 15:31:44.139882 26521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:44.140432 26515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:44.140781 26515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:44.141943 26515 hybrid_clock.cc:648] HybridClock initialized: now 1790263904141941 us; error 38 us; skew 500 ppm
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:44.147753 26515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:44.148178 26515 webserver.cc:460] Webserver started at http://127.0.0.1:33115/ using document root <none> and password file <none>
I20260924 15:31:44.148422 26515 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:44.148500 26515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:44.148651 26515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:44.149761 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "983e25ac19734f35895467784af8abbe"
format_stamp: "Formatted at 2026-09-24 15:31:44 on dist-test-slave-4g6f"
server_key: "9cf44b999df0e825ac3fa95817918dc5"
server_key_iv: "54b753999095751727627ed8ee126c7f"
server_key_version: "encryptionkey@0"
I20260924 15:31:44.150139 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "983e25ac19734f35895467784af8abbe"
format_stamp: "Formatted at 2026-09-24 15:31:44 on dist-test-slave-4g6f"
server_key: "9cf44b999df0e825ac3fa95817918dc5"
server_key_iv: "54b753999095751727627ed8ee126c7f"
server_key_version: "encryptionkey@0"
I20260924 15:31:44.151489 26515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:44.152443 26530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:44.152608 26515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:44.152698 26515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "983e25ac19734f35895467784af8abbe"
format_stamp: "Formatted at 2026-09-24 15:31:44 on dist-test-slave-4g6f"
server_key: "9cf44b999df0e825ac3fa95817918dc5"
server_key_iv: "54b753999095751727627ed8ee126c7f"
server_key_version: "encryptionkey@0"
I20260924 15:31:44.152758 26515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:44.170092 26515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:44.171902 26515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:44.172019 26515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:44.175771 26515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43463
I20260924 15:31:44.175850 26583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43463 every 8 connection(s)
I20260924 15:31:44.176149 26515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:44.177086 26584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:44.179311 26584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe: Bootstrap starting.
I20260924 15:31:44.179951 26584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:44.180238 26584 log.cc:824] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:44.180989 26584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe: No bootstrap required, opened a new log
I20260924 15:31:44.182570 26584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260924 15:31:44.182760 26584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:44.182801 26584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983e25ac19734f35895467784af8abbe, State: Initialized, Role: FOLLOWER
I20260924 15:31:44.182844 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26515
I20260924 15:31:44.182961 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:44.182976 26584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260924 15:31:44.183064 26584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:44.183096 26584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:44.183141 5515 external_mini_cluster.cc:1428] Setting key b6de61b3b7dac20f861583723dbba7ef
I20260924 15:31:44.183145 26584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:44.183774 26584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260924 15:31:44.183913 26584 leader_election.cc:302] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983e25ac19734f35895467784af8abbe; no voters:
I20260924 15:31:44.184103 26584 leader_election.cc:288] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:44.184129 26589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:44.184240 26589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [term 1 LEADER]: Becoming Leader. State: Replica: 983e25ac19734f35895467784af8abbe, State: Running, Role: LEADER
I20260924 15:31:44.184362 26584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:44.184417 26589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } }
I20260924 15:31:44.184989 26589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983e25ac19734f35895467784af8abbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } } }
I20260924 15:31:44.185081 26589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:44.185286 26589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983e25ac19734f35895467784af8abbe. Latest consensus state: current_term: 1 leader_uuid: "983e25ac19734f35895467784af8abbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983e25ac19734f35895467784af8abbe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43463 } } }
I20260924 15:31:44.185374 26598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:44.185413 26589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:44.185940 26598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:44.187577 26598 catalog_manager.cc:1347] Generated new cluster ID: 1ce7ae584ba94e6fa9b529088e00f36c
I20260924 15:31:44.187630 26598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:31:44.191915 26598 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:44.192669 26598 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:44.195348 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:44.183616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:33229 (local address 127.0.0.1:43463)
0924 15:31:44.183804 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:44.183815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:44.183851 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:31:44.183968 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:44.183990 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:44.184064 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:44.184216 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:44.185034 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:44.186448 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:44.187530 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:44.188142 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:44.188219 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:44.191175 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:44.191179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:44.191180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:44.191192 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:44.193663 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:44.194138 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:44.194141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:44.194141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:44.194237 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:44.194490 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:44.194490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:44.194491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:44.194735 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:44.194891 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:44.195045 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:31:44.195140 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260924 15:31:44.196245 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:44.204752 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:44.197586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:35539 (local address 127.0.0.1:43463)
0924 15:31:44.197617 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:44.197618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:44.197845 (+ 227us) server_negotiation.cc:407] Connection header received
0924 15:31:44.197926 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:44.197926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:44.197935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:44.197971 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:44.198592 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:44.199350 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:44.200416 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:44.200728 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:44.200762 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:44.202105 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:44.202107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:44.202107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:44.202113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:44.203531 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:44.203936 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:44.203937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:44.203937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:44.203997 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:44.204205 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:44.204206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:44.204206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:44.204307 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:44.204386 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:44.204614 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:31:44.204678 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260924 15:31:44.205119 26598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983e25ac19734f35895467784af8abbe: Generated new TSK 0
I20260924 15:31:44.205284 26598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:44.209259 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26515
W20260924 15:31:44.213833 26500 connection.cc:535] client connection to 127.0.0.1:43463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43463): Connection reset by peer (error 104)
2026-09-24T15:31:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26616](info): set up 2 sockets
Sep 24 15:31:44 dist-test-slave-4g6f krb5kdc[26616](info): commencing operation
krb5kdc: starting...
W20260924 15:31:45.244817 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:45 dist-test-slave-4g6f krb5kdc[26616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:45.356932 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:45.436766 26630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:45.436959 26630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:45.436990 26630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:45.438133 26630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:45.438174 26630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:45.438189 26630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:45.438239 26630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:45.438265 26630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:45.439745 26630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:45.440094 26630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:45.440351 26630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:45.445250 26636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:45.445341 26635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:45.445242 26638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:45.445482 26630 server_base.cc:1034] running on GCE node
I20260924 15:31:45.445712 26630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:45.446007 26630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:45.447170 26630 hybrid_clock.cc:648] HybridClock initialized: now 1790263905447165 us; error 37 us; skew 500 ppm
Sep 24 15:31:45 dist-test-slave-4g6f krb5kdc[26616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:45.449625 26630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:45.450063 26630 webserver.cc:460] Webserver started at http://127.0.0.1:38793/ using document root <none> and password file <none>
I20260924 15:31:45.450311 26630 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:45.450380 26630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:45.450536 26630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:45.451598 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "4beedf4816cc4f84a157593216ba76bc"
format_stamp: "Formatted at 2026-09-24 15:31:45 on dist-test-slave-4g6f"
server_key: "c05259bceaa218e10d00ca7376c3e8bd"
server_key_iv: "4790604f5ea793755ef9671e3cc1df2b"
server_key_version: "encryptionkey@0"
I20260924 15:31:45.452047 26630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "4beedf4816cc4f84a157593216ba76bc"
format_stamp: "Formatted at 2026-09-24 15:31:45 on dist-test-slave-4g6f"
server_key: "c05259bceaa218e10d00ca7376c3e8bd"
server_key_iv: "4790604f5ea793755ef9671e3cc1df2b"
server_key_version: "encryptionkey@0"
I20260924 15:31:45.453364 26630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 15:31:45.454093 26645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:45.454267 26630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:45.454373 26630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "4beedf4816cc4f84a157593216ba76bc"
format_stamp: "Formatted at 2026-09-24 15:31:45 on dist-test-slave-4g6f"
server_key: "c05259bceaa218e10d00ca7376c3e8bd"
server_key_iv: "4790604f5ea793755ef9671e3cc1df2b"
server_key_version: "encryptionkey@0"
I20260924 15:31:45.454450 26630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:45.464910 26630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:45.466758 26630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:45.466928 26630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:45.470949 26630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34869
I20260924 15:31:45.471027 26698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34869 every 8 connection(s)
I20260924 15:31:45.471361 26630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:45.472395 26699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:45.474668 26699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc: Bootstrap starting.
I20260924 15:31:45.475274 26699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:45.475562 26699 log.cc:824] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:45.476282 26699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc: No bootstrap required, opened a new log
I20260924 15:31:45.477854 26699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } }
I20260924 15:31:45.478001 26699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:45.478025 26699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4beedf4816cc4f84a157593216ba76bc, State: Initialized, Role: FOLLOWER
I20260924 15:31:45.478237 26699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } }
I20260924 15:31:45.478322 26699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:45.478371 26699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:45.478423 26699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:45.478945 26699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } }
I20260924 15:31:45.479066 26699 leader_election.cc:302] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4beedf4816cc4f84a157593216ba76bc; no voters:
I20260924 15:31:45.479262 26699 leader_election.cc:288] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:45.479321 26702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:45.479514 26702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [term 1 LEADER]: Becoming Leader. State: Replica: 4beedf4816cc4f84a157593216ba76bc, State: Running, Role: LEADER
I20260924 15:31:45.479528 26699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:45.479611 26702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } }
I20260924 15:31:45.480304 26704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4beedf4816cc4f84a157593216ba76bc. Latest consensus state: current_term: 1 leader_uuid: "4beedf4816cc4f84a157593216ba76bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } } }
I20260924 15:31:45.480311 26703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4beedf4816cc4f84a157593216ba76bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beedf4816cc4f84a157593216ba76bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34869 } } }
I20260924 15:31:45.480394 26704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:45.480418 26703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:45.480590 26711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:45.481151 26711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:45.481232 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26630
I20260924 15:31:45.481349 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:45.481560 5515 external_mini_cluster.cc:1428] Setting key ea787396c08832cb272ae0595ce9c297
I20260924 15:31:45.482811 26711 catalog_manager.cc:1347] Generated new cluster ID: ef0c4a55e9e74aca9079c6a2109cc698
I20260924 15:31:45.482905 26711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:45.488106 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:45.482012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:38341 (local address 127.0.0.1:34869)
0924 15:31:45.482531 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:45.482541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:45.482573 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:31:45.482688 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:45.482696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:45.482757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:45.482897 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:45.483146 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:45.484623 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:45.485696 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:45.486353 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:45.486466 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:45.487194 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:45.487198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:45.487198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:45.487411 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:45.487450 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:45.487466 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:45.487567 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:45.487651 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:31:45.487714 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":72,"threads_started":1}
I20260924 15:31:45.489245 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:45.491083 26711 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:45.491526 26711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:45.494999 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:45.490735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:50029 (local address 127.0.0.1:34869)
0924 15:31:45.490785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:45.490787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:45.490895 (+ 108us) server_negotiation.cc:407] Connection header received
0924 15:31:45.490958 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:45.490959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:45.490969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:45.491019 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:45.491914 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:45.492663 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:45.493650 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:45.493978 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:45.494053 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:45.494725 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:45.494726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:45.494727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:45.494731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:45.494746 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:45.494751 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:45.494855 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:45.494914 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:31:45.494925 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:31:45.503087 26711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4beedf4816cc4f84a157593216ba76bc: Generated new TSK 0
I20260924 15:31:45.503265 26711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:45.505347 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26630
W20260924 15:31:45.509747 26614 connection.cc:535] client connection to 127.0.0.1:34869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34869): Connection reset by peer (error 104)
2026-09-24T15:31:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:45 dist-test-slave-4g6f krb5kdc[26730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:45 dist-test-slave-4g6f krb5kdc[26730](info): set up 2 sockets
Sep 24 15:31:45 dist-test-slave-4g6f krb5kdc[26730](info): commencing operation
krb5kdc: starting...
W20260924 15:31:46.532832 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:46 dist-test-slave-4g6f krb5kdc[26730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:46.649984 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:46.730414 26743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:46.730609 26743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:46.730630 26743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:46.731904 26743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:46.731945 26743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:46.731966 26743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:46.731979 26743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:46.732028 26743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:46.733496 26743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:46.733741 26743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:46.733969 26743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:46.738840 26750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:46.738844 26752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:46.738883 26749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:46.739073 26743 server_base.cc:1034] running on GCE node
I20260924 15:31:46.739313 26743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:46.739589 26743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:46.740762 26743 hybrid_clock.cc:648] HybridClock initialized: now 1790263906740775 us; error 54 us; skew 500 ppm
Sep 24 15:31:46 dist-test-slave-4g6f krb5kdc[26730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:46.743168 26743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:46.743629 26743 webserver.cc:460] Webserver started at http://127.0.0.1:40735/ using document root <none> and password file <none>
I20260924 15:31:46.743876 26743 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:46.743918 26743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:46.744011 26743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:46.744993 26743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "6d8461c857ef4618b76d37eec99f452a"
format_stamp: "Formatted at 2026-09-24 15:31:46 on dist-test-slave-4g6f"
server_key: "c3edf5fbd733dfe4fcf0e68a213d0542"
server_key_iv: "01c9e1d3aec25e475a906e13ac4571dd"
server_key_version: "encryptionkey@0"
I20260924 15:31:46.745298 26743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "6d8461c857ef4618b76d37eec99f452a"
format_stamp: "Formatted at 2026-09-24 15:31:46 on dist-test-slave-4g6f"
server_key: "c3edf5fbd733dfe4fcf0e68a213d0542"
server_key_iv: "01c9e1d3aec25e475a906e13ac4571dd"
server_key_version: "encryptionkey@0"
I20260924 15:31:46.746515 26743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:46.747133 26759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:46.747313 26743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:46.747421 26743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "6d8461c857ef4618b76d37eec99f452a"
format_stamp: "Formatted at 2026-09-24 15:31:46 on dist-test-slave-4g6f"
server_key: "c3edf5fbd733dfe4fcf0e68a213d0542"
server_key_iv: "01c9e1d3aec25e475a906e13ac4571dd"
server_key_version: "encryptionkey@0"
I20260924 15:31:46.747478 26743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:46.750653 26743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:46.752514 26743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:46.752615 26743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:46.756458 26743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46057
I20260924 15:31:46.756525 26812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46057 every 8 connection(s)
I20260924 15:31:46.756742 26743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:46.757656 26813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:46.759922 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a: Bootstrap starting.
I20260924 15:31:46.760493 26813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:46.760720 26813 log.cc:824] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:46.761425 26813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a: No bootstrap required, opened a new log
I20260924 15:31:46.762943 26813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260924 15:31:46.763077 26813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:46.763099 26813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8461c857ef4618b76d37eec99f452a, State: Initialized, Role: FOLLOWER
I20260924 15:31:46.763232 26813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260924 15:31:46.763314 26813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:46.763335 26813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:46.763363 26813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:46.763918 26813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260924 15:31:46.764004 26813 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d8461c857ef4618b76d37eec99f452a; no voters:
I20260924 15:31:46.764125 26813 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:46.764173 26816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:46.764338 26816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [term 1 LEADER]: Becoming Leader. State: Replica: 6d8461c857ef4618b76d37eec99f452a, State: Running, Role: LEADER
I20260924 15:31:46.764389 26813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:46.764509 26816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260924 15:31:46.764638 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26743
I20260924 15:31:46.764739 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:46.764909 5515 external_mini_cluster.cc:1428] Setting key e9c7dfd1fd19f5ced6dacca00b172f68
I20260924 15:31:46.764997 26818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8461c857ef4618b76d37eec99f452a. Latest consensus state: current_term: 1 leader_uuid: "6d8461c857ef4618b76d37eec99f452a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260924 15:31:46.765076 26818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:46.765389 26817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d8461c857ef4618b76d37eec99f452a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8461c857ef4618b76d37eec99f452a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260924 15:31:46.765460 26817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:46.765836 26825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:46.766462 26825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:46.767474 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:46.765513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:52677 (local address 127.0.0.1:46057)
0924 15:31:46.765673 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:46.765683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:46.765907 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:31:46.766030 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:46.766039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:46.766074 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:46.766229 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:46.766981 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:46.766984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:46.766984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:46.766988 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:46.767038 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:46.767049 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:46.767123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:46.767193 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:31:46.767242 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260924 15:31:46.768257 26825 catalog_manager.cc:1347] Generated new cluster ID: 7800e8340128449bbd3a6ca216410373
I20260924 15:31:46.768306 26825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:46.768433 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:46.770951 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:46.769898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:44607 (local address 127.0.0.1:46057)
0924 15:31:46.769931 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:46.769933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:46.770075 (+ 142us) server_negotiation.cc:407] Connection header received
0924 15:31:46.770123 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:46.770124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:46.770130 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:46.770181 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:46.770740 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:46.770742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:46.770742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:46.770747 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:46.770763 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:46.770768 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:46.770828 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:46.770879 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 15:31:46.770892 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:31:46.788488 26825 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:46.788672 26825 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:46.793685 26825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8461c857ef4618b76d37eec99f452a: Generated new TSK 0
I20260924 15:31:46.793864 26825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:46.797835 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26743
2026-09-24T15:31:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:46 dist-test-slave-4g6f krb5kdc[26844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:46 dist-test-slave-4g6f krb5kdc[26844](info): set up 2 sockets
Sep 24 15:31:46 dist-test-slave-4g6f krb5kdc[26844](info): commencing operation
krb5kdc: starting...
W20260924 15:31:47.832944 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:47 dist-test-slave-4g6f krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:47.950073 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:48.029883 26857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:48.030157 26857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:48.030192 26857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:48.031381 26857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:48.031451 26857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:48.031477 26857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:48.031498 26857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:48.031529 26857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:48.033033 26857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:48.033299 26857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:48.033535 26857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:48.038270 26864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:48.038316 26866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:48.038434 26857 server_base.cc:1034] running on GCE node
W20260924 15:31:48.038316 26862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:48.038775 26857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:48.039110 26857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:48.040269 26857 hybrid_clock.cc:648] HybridClock initialized: now 1790263908040248 us; error 35 us; skew 500 ppm
Sep 24 15:31:48 dist-test-slave-4g6f krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:48.042644 26857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:48.043040 26857 webserver.cc:460] Webserver started at http://127.0.0.1:38287/ using document root <none> and password file <none>
I20260924 15:31:48.043239 26857 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:48.043279 26857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:48.043365 26857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:48.044446 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "10854fcccd22489295540cdffcf4faa0"
format_stamp: "Formatted at 2026-09-24 15:31:48 on dist-test-slave-4g6f"
server_key: "7f524c2b9cef8fd5bbbfd3c554ea3664"
server_key_iv: "f18d9ad06813c412849bdf7f9b44a38f"
server_key_version: "encryptionkey@0"
I20260924 15:31:48.044755 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "10854fcccd22489295540cdffcf4faa0"
format_stamp: "Formatted at 2026-09-24 15:31:48 on dist-test-slave-4g6f"
server_key: "7f524c2b9cef8fd5bbbfd3c554ea3664"
server_key_iv: "f18d9ad06813c412849bdf7f9b44a38f"
server_key_version: "encryptionkey@0"
I20260924 15:31:48.046109 26857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:48.046811 26873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:48.046996 26857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:48.047081 26857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "10854fcccd22489295540cdffcf4faa0"
format_stamp: "Formatted at 2026-09-24 15:31:48 on dist-test-slave-4g6f"
server_key: "7f524c2b9cef8fd5bbbfd3c554ea3664"
server_key_iv: "f18d9ad06813c412849bdf7f9b44a38f"
server_key_version: "encryptionkey@0"
I20260924 15:31:48.047138 26857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:48.071909 26857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:48.073844 26857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:48.073967 26857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:48.077924 26857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34581
I20260924 15:31:48.078011 26927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34581 every 8 connection(s)
I20260924 15:31:48.078358 26857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:48.079380 26928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:48.081698 26928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0: Bootstrap starting.
I20260924 15:31:48.082294 26928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:48.082593 26928 log.cc:824] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:48.083295 26928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0: No bootstrap required, opened a new log
I20260924 15:31:48.084647 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26857
I20260924 15:31:48.084743 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:48.084972 5515 external_mini_cluster.cc:1428] Setting key 55786601b6c5a5ff9195f9ef7ec01c4e
I20260924 15:31:48.085065 26928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } }
I20260924 15:31:48.085233 26928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:48.085281 26928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10854fcccd22489295540cdffcf4faa0, State: Initialized, Role: FOLLOWER
I20260924 15:31:48.085441 26928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } }
I20260924 15:31:48.085551 26928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:48.085620 26928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:48.085680 26928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:48.086728 26928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } }
I20260924 15:31:48.086836 26928 leader_election.cc:302] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10854fcccd22489295540cdffcf4faa0; no voters:
I20260924 15:31:48.086984 26928 leader_election.cc:288] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:48.087054 26933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:48.087227 26933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [term 1 LEADER]: Becoming Leader. State: Replica: 10854fcccd22489295540cdffcf4faa0, State: Running, Role: LEADER
I20260924 15:31:48.087235 26928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:48.087387 26933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } }
I20260924 15:31:48.087798 26934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10854fcccd22489295540cdffcf4faa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } } }
I20260924 15:31:48.087798 26935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10854fcccd22489295540cdffcf4faa0. Latest consensus state: current_term: 1 leader_uuid: "10854fcccd22489295540cdffcf4faa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10854fcccd22489295540cdffcf4faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34581 } } }
I20260924 15:31:48.087917 26934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:48.087925 26935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:48.088116 26942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:48.088617 26942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 15:31:48 dist-test-slave-4g6f krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:31:48.091974 26942 catalog_manager.cc:1347] Generated new cluster ID: 62f7c5155c894428a7c54a8cb6526518
I20260924 15:31:48.092020 26942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:48.096354 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:48.085473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:39821 (local address 127.0.0.1:34581)
0924 15:31:48.085677 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:48.085686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:48.085720 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:31:48.085839 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:48.085847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:48.085905 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:48.086085 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:48.086690 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:48.088151 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:48.089192 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:48.089720 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:48.089844 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:48.092486 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:48.092491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:48.092491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:48.092503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:48.094526 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:48.094994 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:48.094997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:48.094998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:48.095165 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:48.095428 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:48.095429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:48.095430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:48.095776 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:48.095955 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:48.096042 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:31:48.096148 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260924 15:31:48.097432 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:48.100651 26942 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:48.101130 26942 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:48.106225 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:48.098828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:49919 (local address 127.0.0.1:34581)
0924 15:31:48.098863 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:48.098865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:48.098978 (+ 113us) server_negotiation.cc:407] Connection header received
0924 15:31:48.099023 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:48.099025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:48.099036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:48.099102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:48.099699 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:48.100465 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:48.101550 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:48.101870 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:48.101936 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:48.103309 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:48.103310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:48.103311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:48.103317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:48.104911 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:48.105331 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:48.105332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:48.105332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:48.105387 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:48.105628 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:48.105628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:48.105629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:48.105739 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:48.105827 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:48.105987 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:31:48.106046 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:31:48.114464 26942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10854fcccd22489295540cdffcf4faa0: Generated new TSK 0
I20260924 15:31:48.114643 26942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:48.118631 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26857
W20260924 15:31:48.123607 26842 connection.cc:535] client connection to 127.0.0.1:34581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34581): Connection reset by peer (error 104)
2026-09-24T15:31:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:48 dist-test-slave-4g6f krb5kdc[26960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:48 dist-test-slave-4g6f krb5kdc[26960](info): set up 2 sockets
Sep 24 15:31:48 dist-test-slave-4g6f krb5kdc[26960](info): commencing operation
krb5kdc: starting...
W20260924 15:31:49.156776 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:49 dist-test-slave-4g6f krb5kdc[26960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:49.272697 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:49.353039 26974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:49.353269 26974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:49.353312 26974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:49.354435 26974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:49.354501 26974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:49.354525 26974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:49.354545 26974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:49.354573 26974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:49.356096 26974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:49.356377 26974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:49.356614 26974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:49.361514 26979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:49.361565 26982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:49.361629 26974 server_base.cc:1034] running on GCE node
W20260924 15:31:49.361541 26980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:49.362042 26974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:49.362385 26974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:49.363567 26974 hybrid_clock.cc:648] HybridClock initialized: now 1790263909363551 us; error 26 us; skew 500 ppm
Sep 24 15:31:49 dist-test-slave-4g6f krb5kdc[26960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:49.370601 26974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:49.371054 26974 webserver.cc:460] Webserver started at http://127.0.0.1:43835/ using document root <none> and password file <none>
I20260924 15:31:49.371304 26974 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:49.371373 26974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:49.371536 26974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:49.372649 26974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "8b6400771daa4a88bff7788e3fd154f1"
format_stamp: "Formatted at 2026-09-24 15:31:49 on dist-test-slave-4g6f"
server_key: "0c274b4236267516c36ae6a565449372"
server_key_iv: "618a1fffd80a9d8875b999bf00531abd"
server_key_version: "encryptionkey@0"
I20260924 15:31:49.373023 26974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "8b6400771daa4a88bff7788e3fd154f1"
format_stamp: "Formatted at 2026-09-24 15:31:49 on dist-test-slave-4g6f"
server_key: "0c274b4236267516c36ae6a565449372"
server_key_iv: "618a1fffd80a9d8875b999bf00531abd"
server_key_version: "encryptionkey@0"
I20260924 15:31:49.374353 26974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:31:49.375116 26989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:49.375290 26974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:49.375391 26974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "8b6400771daa4a88bff7788e3fd154f1"
format_stamp: "Formatted at 2026-09-24 15:31:49 on dist-test-slave-4g6f"
server_key: "0c274b4236267516c36ae6a565449372"
server_key_iv: "618a1fffd80a9d8875b999bf00531abd"
server_key_version: "encryptionkey@0"
I20260924 15:31:49.375497 26974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:49.394906 26974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:49.396826 26974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:49.397001 26974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:49.401108 26974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41945
I20260924 15:31:49.401144 27041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41945 every 8 connection(s)
I20260924 15:31:49.401508 26974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:49.402411 27042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:49.404821 27042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1: Bootstrap starting.
I20260924 15:31:49.405416 27042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:49.405700 27042 log.cc:824] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:49.406435 27042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1: No bootstrap required, opened a new log
I20260924 15:31:49.407258 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 26974
I20260924 15:31:49.407434 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:49.407624 5515 external_mini_cluster.cc:1428] Setting key 260d61681c0c5f3ce940cc8f4f6eb958
I20260924 15:31:49.408561 27042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260924 15:31:49.408736 27042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:49.408779 27042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6400771daa4a88bff7788e3fd154f1, State: Initialized, Role: FOLLOWER
I20260924 15:31:49.408922 27042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260924 15:31:49.409016 27042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:49.409122 27042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:49.409196 27042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:49.409922 27042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260924 15:31:49.410048 27042 leader_election.cc:302] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b6400771daa4a88bff7788e3fd154f1; no voters:
I20260924 15:31:49.410183 27042 leader_election.cc:288] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:49.410295 27047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:49.410415 27042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:49.410497 27047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b6400771daa4a88bff7788e3fd154f1, State: Running, Role: LEADER
I20260924 15:31:49.410588 27047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } }
I20260924 15:31:49.410948 27048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b6400771daa4a88bff7788e3fd154f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } } }
I20260924 15:31:49.411041 27048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:49.411259 27057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:49.411259 27049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b6400771daa4a88bff7788e3fd154f1. Latest consensus state: current_term: 1 leader_uuid: "8b6400771daa4a88bff7788e3fd154f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b6400771daa4a88bff7788e3fd154f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41945 } } }
I20260924 15:31:49.411362 27049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:49.411903 27057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:49.413932 27057 catalog_manager.cc:1347] Generated new cluster ID: 0cb00db985624dab8b226c30de273eac
I20260924 15:31:49.413983 27057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:49.414297 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:49.408215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:38353 (local address 127.0.0.1:41945)
0924 15:31:49.408432 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:49.408441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:49.408490 (+ 49us) server_negotiation.cc:407] Connection header received
0924 15:31:49.408610 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:49.408617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:49.408674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:49.408834 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:49.409095 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:49.410855 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:49.412111 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:49.412711 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:49.412824 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:49.413583 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:49.413587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:49.413587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:49.413763 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:49.413824 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:49.413841 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:49.413925 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:49.414094 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 15:31:49.414126 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260924 15:31:49.415238 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:49.422973 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:49.419991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:53761 (local address 127.0.0.1:41945)
0924 15:31:49.420029 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:49.420031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:49.420165 (+ 134us) server_negotiation.cc:407] Connection header received
0924 15:31:49.420238 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:49.420239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:49.420250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:49.420323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:49.420499 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:49.420982 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:49.421736 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:49.421942 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:49.422024 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:49.422690 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:49.422691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:49.422692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:49.422696 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:49.422711 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:49.422715 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:49.422789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:49.422856 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:31:49.422867 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:49.423620 27057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:49.424117 27057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:49.430380 27057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b6400771daa4a88bff7788e3fd154f1: Generated new TSK 0
I20260924 15:31:49.430565 27057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:49.436323 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 26974
W20260924 15:31:49.440946 26958 connection.cc:535] client connection to 127.0.0.1:41945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41945): Connection reset by peer (error 104)
2026-09-24T15:31:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:49 dist-test-slave-4g6f krb5kdc[27074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:49 dist-test-slave-4g6f krb5kdc[27074](info): set up 2 sockets
Sep 24 15:31:49 dist-test-slave-4g6f krb5kdc[27074](info): commencing operation
krb5kdc: starting...
W20260924 15:31:50.472956 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:50 dist-test-slave-4g6f krb5kdc[27074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:31:50.590701 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:50.673750 27088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:50.674026 27088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:50.674069 27088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:50.675293 27088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:50.675366 27088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:50.675390 27088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:50.675411 27088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:50.675441 27088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:50.676994 27088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:50.677292 27088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:50.677542 27088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:50.682488 27094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:50.682547 27095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:50.682495 27097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:50.682708 27088 server_base.cc:1034] running on GCE node
I20260924 15:31:50.682974 27088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:50.683223 27088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:50.684367 27088 hybrid_clock.cc:648] HybridClock initialized: now 1790263910684352 us; error 30 us; skew 500 ppm
Sep 24 15:31:50 dist-test-slave-4g6f krb5kdc[27074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:50.686893 27088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:31:50.687315 27088 webserver.cc:460] Webserver started at http://127.0.0.1:37197/ using document root <none> and password file <none>
I20260924 15:31:50.687547 27088 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:50.687616 27088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:50.687795 27088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:50.688897 27088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81"
format_stamp: "Formatted at 2026-09-24 15:31:50 on dist-test-slave-4g6f"
server_key: "1e3d176775b4b1f2364eb3de1c30dc39"
server_key_iv: "682b508676cd9f016521e7b3a55d0477"
server_key_version: "encryptionkey@0"
I20260924 15:31:50.689275 27088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81"
format_stamp: "Formatted at 2026-09-24 15:31:50 on dist-test-slave-4g6f"
server_key: "1e3d176775b4b1f2364eb3de1c30dc39"
server_key_iv: "682b508676cd9f016521e7b3a55d0477"
server_key_version: "encryptionkey@0"
I20260924 15:31:50.690615 27088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:31:50.691332 27105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:50.691506 27088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:50.691619 27088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81"
format_stamp: "Formatted at 2026-09-24 15:31:50 on dist-test-slave-4g6f"
server_key: "1e3d176775b4b1f2364eb3de1c30dc39"
server_key_iv: "682b508676cd9f016521e7b3a55d0477"
server_key_version: "encryptionkey@0"
I20260924 15:31:50.691733 27088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:50.695053 27088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:50.697388 27088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:50.697556 27088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:50.701611 27088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43763
I20260924 15:31:50.701642 27157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43763 every 8 connection(s)
I20260924 15:31:50.701972 27088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:50.702931 27158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:50.705138 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27088
I20260924 15:31:50.705263 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:50.705449 5515 external_mini_cluster.cc:1428] Setting key 34173d4d5f9e9bd81c6499f4361af613
I20260924 15:31:50.705494 27158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81: Bootstrap starting.
I20260924 15:31:50.706142 27158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:50.706434 27158 log.cc:824] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:50.707142 27158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81: No bootstrap required, opened a new log
I20260924 15:31:50.708137 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:50.705954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:37451 (local address 127.0.0.1:43763)
0924 15:31:50.706171 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:50.706183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:50.706220 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:31:50.706823 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:50.706833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:50.706871 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:50.707094 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:50.707725 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:50.707729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:50.707729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:50.707734 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:50.707773 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:50.707786 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:50.707867 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:50.707925 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 15:31:50.707969 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260924 15:31:50.709210 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:50.709743 27158 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260924 15:31:50.709903 27158 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:50.709949 27158 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fbaf7cc0fc4b09b537b344f4c9df81, State: Initialized, Role: FOLLOWER
I20260924 15:31:50.710072 27158 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260924 15:31:50.710253 27158 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:50.710294 27158 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:50.710392 27158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:50.711184 27158 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260924 15:31:50.711319 27158 leader_election.cc:302] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0fbaf7cc0fc4b09b537b344f4c9df81; no voters:
I20260924 15:31:50.711498 27158 leader_election.cc:288] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:50.711551 27168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:50.711730 27168 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [term 1 LEADER]: Becoming Leader. State: Replica: e0fbaf7cc0fc4b09b537b344f4c9df81, State: Running, Role: LEADER
I20260924 15:31:50.711772 27158 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:50.711875 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:50.710660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.134:37455 (local address 127.0.0.1:43763)
0924 15:31:50.710699 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:50.710701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:50.710838 (+ 137us) server_negotiation.cc:407] Connection header received
0924 15:31:50.710913 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:50.710914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:50.710918 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:50.710965 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:50.711507 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:50.711508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:50.711508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:50.711513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:50.711529 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:50.711533 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:50.711591 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:50.711735 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 15:31:50.711750 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:31:50.711896 27168 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260924 15:31:50.712368 27169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } } }
I20260924 15:31:50.712390 27170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fbaf7cc0fc4b09b537b344f4c9df81. Latest consensus state: current_term: 1 leader_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fbaf7cc0fc4b09b537b344f4c9df81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } } }
I20260924 15:31:50.712467 27169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:50.712491 27170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:50.712684 27175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:50.713200 27175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:50.714891 27175 catalog_manager.cc:1347] Generated new cluster ID: 4910513216ae4d72be16f194f4e58141
I20260924 15:31:50.714943 27175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:50.724630 27175 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:50.724809 27175 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:50.727648 27175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0fbaf7cc0fc4b09b537b344f4c9df81: Generated new TSK 0
I20260924 15:31:50.727850 27175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:50.734706 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27088
2026-09-24T15:31:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:50 dist-test-slave-4g6f krb5kdc[27189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:50 dist-test-slave-4g6f krb5kdc[27189](info): set up 2 sockets
Sep 24 15:31:50 dist-test-slave-4g6f krb5kdc[27189](info): commencing operation
krb5kdc: starting...
W20260924 15:31:51.772748 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:51 dist-test-slave-4g6f krb5kdc[27189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:51.890637 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42125
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:51.970392 27204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:51.970595 27204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:51.970618 27204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:51.971731 27204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:51.971767 27204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:51.971781 27204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:51.971794 27204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:51.971807 27204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:51.973232 27204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:51.973421 27204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:51.973584 27204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:51.978555 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:51.978560 27210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:51.978556 27209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:51.978724 27204 server_base.cc:1034] running on GCE node
I20260924 15:31:51.979063 27204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:51.979390 27204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:51.980551 27204 hybrid_clock.cc:648] HybridClock initialized: now 1790263911980526 us; error 41 us; skew 500 ppm
Sep 24 15:31:51 dist-test-slave-4g6f krb5kdc[27189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263911, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:51.983038 27204 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:51.983523 27204 webserver.cc:460] Webserver started at http://127.5.98.254:37593/ using document root <none> and password file <none>
I20260924 15:31:51.983777 27204 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:51.983829 27204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:51.983922 27204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:51.985042 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "80ec2604526e441e82453f963d37363d"
format_stamp: "Formatted at 2026-09-24 15:31:51 on dist-test-slave-4g6f"
server_key: "82535e331d925894db3062d950b00129"
server_key_iv: "47395aad774c54db08b2af3c782ba572"
server_key_version: "encryptionkey@0"
I20260924 15:31:51.985369 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "80ec2604526e441e82453f963d37363d"
format_stamp: "Formatted at 2026-09-24 15:31:51 on dist-test-slave-4g6f"
server_key: "82535e331d925894db3062d950b00129"
server_key_iv: "47395aad774c54db08b2af3c782ba572"
server_key_version: "encryptionkey@0"
I20260924 15:31:51.986649 27204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:51.987427 27219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:51.987638 27204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:51.987756 27204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "80ec2604526e441e82453f963d37363d"
format_stamp: "Formatted at 2026-09-24 15:31:51 on dist-test-slave-4g6f"
server_key: "82535e331d925894db3062d950b00129"
server_key_iv: "47395aad774c54db08b2af3c782ba572"
server_key_version: "encryptionkey@0"
I20260924 15:31:51.987813 27204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:52.001680 27204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:52.003970 27204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:52.004079 27204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:52.008064 27204 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:42125
I20260924 15:31:52.008121 27272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:42125 every 8 connection(s)
I20260924 15:31:52.008458 27204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:52.009341 27273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:52.011583 27273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: Bootstrap starting.
I20260924 15:31:52.012187 27273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:52.012471 27273 log.cc:824] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:52.013164 27273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: No bootstrap required, opened a new log
I20260924 15:31:52.014745 27273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } }
I20260924 15:31:52.014907 27273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:52.014958 27273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ec2604526e441e82453f963d37363d, State: Initialized, Role: FOLLOWER
I20260924 15:31:52.015147 27273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } }
I20260924 15:31:52.015235 27273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:52.015278 27273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:52.015329 27273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:52.015363 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27204
I20260924 15:31:52.015493 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:52.015758 5515 external_mini_cluster.cc:1428] Setting key a879741937b872bef11a48f37a9a2b03
I20260924 15:31:52.016500 27273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } }
I20260924 15:31:52.016614 27273 leader_election.cc:302] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ec2604526e441e82453f963d37363d; no voters:
I20260924 15:31:52.016772 27273 leader_election.cc:288] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:52.016865 27278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:52.017028 27278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [term 1 LEADER]: Becoming Leader. State: Replica: 80ec2604526e441e82453f963d37363d, State: Running, Role: LEADER
I20260924 15:31:52.017057 27273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:52.017148 27278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } }
I20260924 15:31:52.017501 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80ec2604526e441e82453f963d37363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } } }
I20260924 15:31:52.017589 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:52.017740 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ec2604526e441e82453f963d37363d. Latest consensus state: current_term: 1 leader_uuid: "80ec2604526e441e82453f963d37363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ec2604526e441e82453f963d37363d" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42125 } } }
I20260924 15:31:52.017831 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d [sys.catalog]: This master's current role is: LEADER
W20260924 15:31:52.018961 27294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:31:52.019539 27294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:31:52.019621 27286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:52.019825 27286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:52.021351 27286 catalog_manager.cc:1347] Generated new cluster ID: 491705a7bcfe4dd29cb8ed3c4aae97f1
I20260924 15:31:52.021422 27286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:52 dist-test-slave-4g6f krb5kdc[27189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:52.030352 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:52.016326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34491 (local address 127.5.98.254:42125)
0924 15:31:52.016540 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:52.016550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:52.016587 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:31:52.016708 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:52.016716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:52.016775 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:52.016924 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:52.017891 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:52.019258 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:52.020309 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:52.020757 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:52.020891 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:52.026065 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:52.026069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:52.026069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:52.026081 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:52.028541 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:52.029044 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:52.029047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:52.029047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:52.029162 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:52.029426 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:52.029427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:52.029427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:52.029715 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:52.029878 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:52.030038 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:31:52.030131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260924 15:31:52.031760 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:52.034291 27286 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:52.035277 27286 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:52.045198 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:52.033361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55005 (local address 127.5.98.254:42125)
0924 15:31:52.033397 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:52.033399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:52.033753 (+ 354us) server_negotiation.cc:407] Connection header received
0924 15:31:52.033863 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:52.033864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:52.033874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:52.033951 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:52.035466 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:52.036363 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:52.037495 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:52.037779 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:52.037814 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:52.039383 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:52.039385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:52.039385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:52.039391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:52.043817 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:52.044383 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:52.044384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:52.044385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:52.044437 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:52.044655 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:52.044655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:52.044656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:52.044762 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:52.044841 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:52.045048 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 15:31:52.045109 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:31:52.047043 27286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80ec2604526e441e82453f963d37363d: Generated new TSK 0
I20260924 15:31:52.047226 27286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:52.053486 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27204
W20260924 15:31:52.059281 27187 connection.cc:535] client connection to 127.5.98.254:42125 recv error: Network error: failed to read from TLS socket (remote: 127.5.98.254:42125): Connection reset by peer (error 104)
2026-09-24T15:31:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:52 dist-test-slave-4g6f krb5kdc[27304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:52 dist-test-slave-4g6f krb5kdc[27304](info): set up 2 sockets
Sep 24 15:31:52 dist-test-slave-4g6f krb5kdc[27304](info): commencing operation
krb5kdc: starting...
W20260924 15:31:53.088714 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:53 dist-test-slave-4g6f krb5kdc[27304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:53.206377 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34495
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:44009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:34495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:53.286761 27317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:53.287002 27317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:53.287042 27317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:53.288206 27317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:53.288275 27317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:53.288297 27317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:53.288319 27317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:53.288357 27317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:53.289857 27317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:44009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:34495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:34495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:53.290136 27317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:53.290367 27317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:53.295292 27328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:53.295343 27324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:53.295346 27323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:53.295310 27317 server_base.cc:1034] running on GCE node
I20260924 15:31:53.295766 27317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:53.296015 27317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:53.297191 27317 hybrid_clock.cc:648] HybridClock initialized: now 1790263913297171 us; error 37 us; skew 500 ppm
Sep 24 15:31:53 dist-test-slave-4g6f krb5kdc[27304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263913, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:53.299639 27317 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:53.300153 27317 webserver.cc:460] Webserver started at http://127.5.98.254:39207/ using document root <none> and password file <none>
I20260924 15:31:53.300372 27317 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:53.300417 27317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:53.300572 27317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:53.301649 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "572bd639ddfb4e44b6dc75fde84728d4"
format_stamp: "Formatted at 2026-09-24 15:31:53 on dist-test-slave-4g6f"
server_key: "ca6bbbf44be69e2bde896e5209daff47"
server_key_iv: "1fe17da1f93e695113fba3f055ec16c8"
server_key_version: "encryptionkey@0"
I20260924 15:31:53.301954 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "572bd639ddfb4e44b6dc75fde84728d4"
format_stamp: "Formatted at 2026-09-24 15:31:53 on dist-test-slave-4g6f"
server_key: "ca6bbbf44be69e2bde896e5209daff47"
server_key_iv: "1fe17da1f93e695113fba3f055ec16c8"
server_key_version: "encryptionkey@0"
I20260924 15:31:53.303207 27317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:53.303952 27335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:53.304111 27317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:53.304198 27317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "572bd639ddfb4e44b6dc75fde84728d4"
format_stamp: "Formatted at 2026-09-24 15:31:53 on dist-test-slave-4g6f"
server_key: "ca6bbbf44be69e2bde896e5209daff47"
server_key_iv: "1fe17da1f93e695113fba3f055ec16c8"
server_key_version: "encryptionkey@0"
I20260924 15:31:53.304253 27317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:53.321414 27317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:53.323258 27317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:53.323367 27317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:53.327527 27317 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:34495
I20260924 15:31:53.327559 27387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:34495 every 8 connection(s)
I20260924 15:31:53.327926 27317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:53.328773 27388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:53.330981 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27317
I20260924 15:31:53.331080 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:53.331120 27388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: Bootstrap starting.
I20260924 15:31:53.331295 5515 external_mini_cluster.cc:1428] Setting key e04191de61ccb401f4a3447823f0d56d
I20260924 15:31:53.331831 27388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:53.332149 27388 log.cc:824] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:53.332890 27388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: No bootstrap required, opened a new log
I20260924 15:31:53.334986 27388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } }
I20260924 15:31:53.335172 27388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:53.335222 27388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572bd639ddfb4e44b6dc75fde84728d4, State: Initialized, Role: FOLLOWER
I20260924 15:31:53.335361 27388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } }
I20260924 15:31:53.335423 27388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:53.335479 27388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:53.335530 27388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:53.336268 27388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } }
I20260924 15:31:53.336371 27388 leader_election.cc:302] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 572bd639ddfb4e44b6dc75fde84728d4; no voters:
I20260924 15:31:53.336556 27388 leader_election.cc:288] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:53.336581 27393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:53.336763 27393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [term 1 LEADER]: Becoming Leader. State: Replica: 572bd639ddfb4e44b6dc75fde84728d4, State: Running, Role: LEADER
I20260924 15:31:53.336798 27388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:53.336860 27393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } }
I20260924 15:31:53.337200 27393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 572bd639ddfb4e44b6dc75fde84728d4. Latest consensus state: current_term: 1 leader_uuid: "572bd639ddfb4e44b6dc75fde84728d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } } }
I20260924 15:31:53.337292 27393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:53.337577 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:53.331807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42787 (local address 127.5.98.254:34495)
0924 15:31:53.332007 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:53.332040 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:53.332066 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:31:53.332701 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:53.332709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:53.332758 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:53.332873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:53.333128 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:53.333983 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:53.335204 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:53.335869 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:53.335951 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:53.336713 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:53.336717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:53.336717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:53.336922 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:53.336963 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:53.336982 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:53.337067 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:53.337180 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 15:31:53.337209 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260924 15:31:53.337894 27395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "572bd639ddfb4e44b6dc75fde84728d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572bd639ddfb4e44b6dc75fde84728d4" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 34495 } } }
I20260924 15:31:53.337961 27395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:53.338560 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 15:31:53.340091 27409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:31:53.340149 27409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:31:53.340220 27400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:53.340445 27400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:53.341781 27400 catalog_manager.cc:1347] Generated new cluster ID: 49a0e5a8fd604c698da65604124705a8
I20260924 15:31:53.341841 27400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:53.343197 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:53.339852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59375 (local address 127.5.98.254:34495)
0924 15:31:53.339891 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:53.339893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:53.340008 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:31:53.340037 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:53.340038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:53.340052 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:53.340113 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:53.340280 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:53.340848 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:53.341921 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:53.342225 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:53.342292 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:53.342946 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:53.342947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:53.342948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:53.342951 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:53.342965 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:53.342969 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:53.343030 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:53.343122 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:31:53.343132 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:31:53.360210 27400 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:53.360781 27400 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:53.363979 27400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 572bd639ddfb4e44b6dc75fde84728d4: Generated new TSK 0
I20260924 15:31:53.364147 27400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:53.372337 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27317
W20260924 15:31:53.376961 27302 connection.cc:535] client connection to 127.5.98.254:34495 recv error: Network error: failed to read from TLS socket (remote: 127.5.98.254:34495): Connection reset by peer (error 104)
2026-09-24T15:31:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:53 dist-test-slave-4g6f krb5kdc[27420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:53 dist-test-slave-4g6f krb5kdc[27420](info): set up 2 sockets
Sep 24 15:31:53 dist-test-slave-4g6f krb5kdc[27420](info): commencing operation
krb5kdc: starting...
W20260924 15:31:54.412753 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:54 dist-test-slave-4g6f krb5kdc[27420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:54.530222 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36057
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:39851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:54.609404 27433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:54.609611 27433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:54.609632 27433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:54.610684 27433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:54.610718 27433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:54.610731 27433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:54.610744 27433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:54.610756 27433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:54.612150 27433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:39851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:36057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:54.612339 27433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:54.612507 27433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:54.617357 27439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:54.617352 27438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:54.617352 27441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:54.617621 27433 server_base.cc:1034] running on GCE node
I20260924 15:31:54.617863 27433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:54.618136 27433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:54.619300 27433 hybrid_clock.cc:648] HybridClock initialized: now 1790263914619283 us; error 28 us; skew 500 ppm
Sep 24 15:31:54 dist-test-slave-4g6f krb5kdc[27420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263914, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:54.625710 27433 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:54.626135 27433 webserver.cc:460] Webserver started at http://127.5.98.254:39547/ using document root <none> and password file <none>
I20260924 15:31:54.626377 27433 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:54.626454 27433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:54.626590 27433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:54.627631 27433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "ddf09dc1435143cfab495303a0881340"
format_stamp: "Formatted at 2026-09-24 15:31:54 on dist-test-slave-4g6f"
server_key: "0f18adf8b071bd8686fa8a7fad210c98"
server_key_iv: "e88252f0e18f624f8165bb763e98516b"
server_key_version: "encryptionkey@0"
I20260924 15:31:54.628062 27433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "ddf09dc1435143cfab495303a0881340"
format_stamp: "Formatted at 2026-09-24 15:31:54 on dist-test-slave-4g6f"
server_key: "0f18adf8b071bd8686fa8a7fad210c98"
server_key_iv: "e88252f0e18f624f8165bb763e98516b"
server_key_version: "encryptionkey@0"
I20260924 15:31:54.629297 27433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:54.629983 27448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:54.630146 27433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:54.630239 27433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "ddf09dc1435143cfab495303a0881340"
format_stamp: "Formatted at 2026-09-24 15:31:54 on dist-test-slave-4g6f"
server_key: "0f18adf8b071bd8686fa8a7fad210c98"
server_key_iv: "e88252f0e18f624f8165bb763e98516b"
server_key_version: "encryptionkey@0"
I20260924 15:31:54.630323 27433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:54.633538 27433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:54.635777 27433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:54.635926 27433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:54.639621 27433 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:36057
I20260924 15:31:54.639650 27501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:36057 every 8 connection(s)
I20260924 15:31:54.640026 27433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:54.640966 27502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:54.642953 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: Bootstrap starting.
I20260924 15:31:54.643442 27502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:54.643743 27502 log.cc:824] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:54.644161 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27433
I20260924 15:31:54.644251 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:54.644440 5515 external_mini_cluster.cc:1428] Setting key 253287d29a5b97acacd0a055870b26b2
I20260924 15:31:54.644464 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: No bootstrap required, opened a new log
I20260924 15:31:54.646771 27502 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } }
I20260924 15:31:54.646932 27502 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:54.646994 27502 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf09dc1435143cfab495303a0881340, State: Initialized, Role: FOLLOWER
I20260924 15:31:54.647126 27502 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } }
I20260924 15:31:54.647228 27502 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:54.647259 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:54.644965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42877 (local address 127.5.98.254:36057)
0924 15:31:54.645169 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:54.645179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:54.645218 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:31:54.645857 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:54.645866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:54.645901 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:54.646051 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:54.646640 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:54.646644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:54.646644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:54.646648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:54.646691 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:54.646705 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:54.646783 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:54.646836 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 15:31:54.646875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260924 15:31:54.647266 27502 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:54.647337 27502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:54.648120 27502 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } }
I20260924 15:31:54.648177 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:54.648223 27502 leader_election.cc:302] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf09dc1435143cfab495303a0881340; no voters:
I20260924 15:31:54.648401 27502 leader_election.cc:288] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:54.648459 27507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:54.648609 27507 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [term 1 LEADER]: Becoming Leader. State: Replica: ddf09dc1435143cfab495303a0881340, State: Running, Role: LEADER
I20260924 15:31:54.648659 27502 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:54.648705 27507 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } }
I20260924 15:31:54.649008 27507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf09dc1435143cfab495303a0881340. Latest consensus state: current_term: 1 leader_uuid: "ddf09dc1435143cfab495303a0881340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } } }
I20260924 15:31:54.649098 27507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:54.649019 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf09dc1435143cfab495303a0881340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf09dc1435143cfab495303a0881340" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 36057 } } }
I20260924 15:31:54.649268 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340 [sys.catalog]: This master's current role is: LEADER
W20260924 15:31:54.650343 27528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:31:54.650408 27528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:31:54.650731 27514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:54.650907 27514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:54.651013 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:54.649771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40347 (local address 127.5.98.254:36057)
0924 15:31:54.649806 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:54.649808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:54.649942 (+ 134us) server_negotiation.cc:407] Connection header received
0924 15:31:54.650062 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:54.650063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:54.650067 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:54.650106 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:54.650694 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:54.650696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:54.650696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:54.650699 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:54.650711 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:54.650714 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:54.650784 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:54.650878 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:31:54.650893 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:31:54.652485 27514 catalog_manager.cc:1347] Generated new cluster ID: 1bee93182b834c01a0d417f8f84be2b2
I20260924 15:31:54.652527 27514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:54.657198 27514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:54.657347 27514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:54.661175 27514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf09dc1435143cfab495303a0881340: Generated new TSK 0
I20260924 15:31:54.661356 27514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:54.663689 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27433
2026-09-24T15:31:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:54 dist-test-slave-4g6f krb5kdc[27534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:54 dist-test-slave-4g6f krb5kdc[27534](info): set up 2 sockets
Sep 24 15:31:54 dist-test-slave-4g6f krb5kdc[27534](info): commencing operation
krb5kdc: starting...
W20260924 15:31:55.680768 5515 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:55 dist-test-slave-4g6f krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:55.800324 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:46373
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:46373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:55.880429 27548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:55.880646 27548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:55.880686 27548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:55.881793 27548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:55.881863 27548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:55.881891 27548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:55.881919 27548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:55.881948 27548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:55.883395 27548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:46373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:46373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:55.883711 27548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:55.883950 27548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:55.888679 27553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:55.888679 27554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:55.888710 27556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:55.888814 27548 server_base.cc:1034] running on GCE node
I20260924 15:31:55.889113 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:55.889338 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:55.890487 27548 hybrid_clock.cc:648] HybridClock initialized: now 1790263915890472 us; error 33 us; skew 500 ppm
Sep 24 15:31:55 dist-test-slave-4g6f krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263915, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:55.892897 27548 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:55.893342 27548 webserver.cc:460] Webserver started at http://127.5.98.254:38735/ using document root <none> and password file <none>
I20260924 15:31:55.893556 27548 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:55.893601 27548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:55.893744 27548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:55.894747 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "eab3b89df62d4e6185fcaad7e0ae98fc"
format_stamp: "Formatted at 2026-09-24 15:31:55 on dist-test-slave-4g6f"
server_key: "a7bc4d973b526e912f989b42b786450a"
server_key_iv: "225ef9601ccff94dd658a2a37e3a8e16"
server_key_version: "encryptionkey@0"
I20260924 15:31:55.895134 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "eab3b89df62d4e6185fcaad7e0ae98fc"
format_stamp: "Formatted at 2026-09-24 15:31:55 on dist-test-slave-4g6f"
server_key: "a7bc4d973b526e912f989b42b786450a"
server_key_iv: "225ef9601ccff94dd658a2a37e3a8e16"
server_key_version: "encryptionkey@0"
I20260924 15:31:55.896551 27548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:55.897419 27563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:55.897728 27548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:31:55.897850 27548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "eab3b89df62d4e6185fcaad7e0ae98fc"
format_stamp: "Formatted at 2026-09-24 15:31:55 on dist-test-slave-4g6f"
server_key: "a7bc4d973b526e912f989b42b786450a"
server_key_iv: "225ef9601ccff94dd658a2a37e3a8e16"
server_key_version: "encryptionkey@0"
I20260924 15:31:55.897926 27548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:55.909138 27548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:55.910810 27548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:55.910965 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:55.914994 27548 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:46373
I20260924 15:31:55.915046 27617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:46373 every 8 connection(s)
I20260924 15:31:55.915373 27548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:55.916157 27618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:55.918303 27618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc: Bootstrap starting.
I20260924 15:31:55.918872 27618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:55.919155 27618 log.cc:824] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:55.919888 27618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc: No bootstrap required, opened a new log
I20260924 15:31:55.921424 27618 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } }
I20260924 15:31:55.921638 27618 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:55.921692 27618 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab3b89df62d4e6185fcaad7e0ae98fc, State: Initialized, Role: FOLLOWER
I20260924 15:31:55.921859 27618 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } }
I20260924 15:31:55.921950 27618 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:55.921980 27618 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:55.922027 27618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:55.923004 27618 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } }
I20260924 15:31:55.923117 27618 leader_election.cc:302] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab3b89df62d4e6185fcaad7e0ae98fc; no voters:
I20260924 15:31:55.923274 27618 leader_election.cc:288] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:55.923327 27621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:55.923532 27618 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:55.923529 27621 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [term 1 LEADER]: Becoming Leader. State: Replica: eab3b89df62d4e6185fcaad7e0ae98fc, State: Running, Role: LEADER
I20260924 15:31:55.923681 27621 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } }
I20260924 15:31:55.924057 27622 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } } }
I20260924 15:31:55.924144 27622 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:55.924045 27623 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab3b89df62d4e6185fcaad7e0ae98fc. Latest consensus state: current_term: 1 leader_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab3b89df62d4e6185fcaad7e0ae98fc" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 46373 } } }
I20260924 15:31:55.924280 27623 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:55.924508 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27548
I20260924 15:31:55.924621 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:55.924782 27631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:55.924819 5515 external_mini_cluster.cc:1428] Setting key 8d9667bd117844bb05b2b1689dac6f20
I20260924 15:31:55.925534 27631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:55.927073 27631 catalog_manager.cc:1347] Generated new cluster ID: 3edf2e3bc9614dfcb578be607a996bfd
I20260924 15:31:55.927129 27631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:31:55 dist-test-slave-4g6f krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:55.935123 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:55.925266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58945 (local address 127.5.98.254:46373)
0924 15:31:55.925452 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:55.925461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:55.925526 (+ 65us) server_negotiation.cc:407] Connection header received
0924 15:31:55.925702 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:55.925710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:55.925776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:55.925929 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:55.926676 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:55.927793 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:55.928575 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:55.929001 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:55.929118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:55.931384 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:55.931388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:55.931388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:55.931399 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:55.933553 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:55.933955 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:55.933964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:55.933965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:55.934056 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:55.934263 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:55.934264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:55.934265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:55.934485 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:55.934659 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:55.934770 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 15:31:55.934913 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260924 15:31:55.936038 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:55.936872 27631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:55.937548 27631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:55.945700 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:55.937496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48465 (local address 127.5.98.254:46373)
0924 15:31:55.937532 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:55.937534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:55.939872 (+ 2338us) server_negotiation.cc:407] Connection header received
0924 15:31:55.939961 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:55.939962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:55.939973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:55.940021 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:55.940546 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:55.941026 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:55.941779 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:55.941990 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:55.942073 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:55.943176 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:55.943177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:55.943178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:55.943184 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:55.944567 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:55.944938 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:55.944939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:55.944940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:55.944998 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:55.945200 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:55.945201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:55.945201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:55.945310 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:55.945405 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:55.945535 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 15:31:55.945605 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:55.950632 27631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab3b89df62d4e6185fcaad7e0ae98fc: Generated new TSK 0
I20260924 15:31:55.950788 27631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:55.956970 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27548
2026-09-24T15:31:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:56 dist-test-slave-4g6f krb5kdc[27650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:56 dist-test-slave-4g6f krb5kdc[27650](info): set up 2 sockets
Sep 24 15:31:56 dist-test-slave-4g6f krb5kdc[27650](info): commencing operation
krb5kdc: starting...
W20260924 15:31:56.992868 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:57 dist-test-slave-4g6f krb5kdc[27650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:57.106238 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40303
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:43081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:57.185055 27663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:57.185268 27663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:57.185305 27663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:57.186440 27663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:57.186513 27663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:57.186537 27663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:57.186568 27663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:57.186599 27663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:57.188069 27663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:43081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:40303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:57.188334 27663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:57.188560 27663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:31:57.193496 27663 server_base.cc:1034] running on GCE node
W20260924 15:31:57.193415 27669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:57.193395 27671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:57.193437 27668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:57.193899 27663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:57.194209 27663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:57.195362 27663 hybrid_clock.cc:648] HybridClock initialized: now 1790263917195337 us; error 40 us; skew 500 ppm
Sep 24 15:31:57 dist-test-slave-4g6f krb5kdc[27650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263917, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:57.197788 27663 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:57.198175 27663 webserver.cc:460] Webserver started at http://127.5.98.254:33697/ using document root <none> and password file <none>
I20260924 15:31:57.198408 27663 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:57.198474 27663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:57.198625 27663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:57.199628 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "27309e75f1934eec9e8fbf4011276a14"
format_stamp: "Formatted at 2026-09-24 15:31:57 on dist-test-slave-4g6f"
server_key: "1b2bbb678e8a91a27f32ac0d47b97b48"
server_key_iv: "d189537a8e3496a73acc644be63c62e2"
server_key_version: "encryptionkey@0"
I20260924 15:31:57.200057 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "27309e75f1934eec9e8fbf4011276a14"
format_stamp: "Formatted at 2026-09-24 15:31:57 on dist-test-slave-4g6f"
server_key: "1b2bbb678e8a91a27f32ac0d47b97b48"
server_key_iv: "d189537a8e3496a73acc644be63c62e2"
server_key_version: "encryptionkey@0"
I20260924 15:31:57.201471 27663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:57.202216 27679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:57.202422 27663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:31:57.202523 27663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "27309e75f1934eec9e8fbf4011276a14"
format_stamp: "Formatted at 2026-09-24 15:31:57 on dist-test-slave-4g6f"
server_key: "1b2bbb678e8a91a27f32ac0d47b97b48"
server_key_iv: "d189537a8e3496a73acc644be63c62e2"
server_key_version: "encryptionkey@0"
I20260924 15:31:57.202608 27663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:57.213635 27663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:57.215803 27663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:57.215960 27663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:57.219596 27663 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:40303
I20260924 15:31:57.219646 27732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:40303 every 8 connection(s)
I20260924 15:31:57.220072 27663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:57.220973 27733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:57.223143 27733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14: Bootstrap starting.
I20260924 15:31:57.223750 27733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:57.224040 27733 log.cc:824] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:57.224706 27733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14: No bootstrap required, opened a new log
I20260924 15:31:57.226217 27733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } }
I20260924 15:31:57.226378 27733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:57.226403 27733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27309e75f1934eec9e8fbf4011276a14, State: Initialized, Role: FOLLOWER
I20260924 15:31:57.226567 27733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } }
I20260924 15:31:57.226627 27733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:57.226647 27733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:57.226714 27733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:57.227541 27733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } }
I20260924 15:31:57.227650 27733 leader_election.cc:302] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27309e75f1934eec9e8fbf4011276a14; no voters:
I20260924 15:31:57.227843 27733 leader_election.cc:288] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:57.227891 27736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:57.228044 27736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [term 1 LEADER]: Becoming Leader. State: Replica: 27309e75f1934eec9e8fbf4011276a14, State: Running, Role: LEADER
I20260924 15:31:57.228113 27733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:57.228206 27736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } }
I20260924 15:31:57.228567 27737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27309e75f1934eec9e8fbf4011276a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } } }
I20260924 15:31:57.228574 27738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27309e75f1934eec9e8fbf4011276a14. Latest consensus state: current_term: 1 leader_uuid: "27309e75f1934eec9e8fbf4011276a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27309e75f1934eec9e8fbf4011276a14" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 40303 } } }
I20260924 15:31:57.228665 27737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:57.228693 27738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14 [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:57.228922 27744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:57.229491 27744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:57.230206 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27663
I20260924 15:31:57.230325 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:57.230506 5515 external_mini_cluster.cc:1428] Setting key 3101914da4a0bb88551886276d935162
I20260924 15:31:57.231441 27744 catalog_manager.cc:1347] Generated new cluster ID: 77dbdce109784118904a6efdc2b95068
I20260924 15:31:57.231485 27744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:57.236371 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:57.230893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32775 (local address 127.5.98.254:40303)
0924 15:31:57.231067 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:57.231074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:57.231156 (+ 82us) server_negotiation.cc:407] Connection header received
0924 15:31:57.231289 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:57.231296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:57.231351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:57.231484 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:57.231721 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:57.233068 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:57.234204 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:57.234788 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:57.234893 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:57.235553 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:57.235557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:57.235557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:57.235792 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:57.235828 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:57.235842 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:57.235916 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:57.236032 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:31:57.236090 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260924 15:31:57.237082 27744 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:57.237246 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:57.237737 27744 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:57.241439 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:57.238586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45199 (local address 127.5.98.254:40303)
0924 15:31:57.238631 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:57.238633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:57.238762 (+ 129us) server_negotiation.cc:407] Connection header received
0924 15:31:57.238831 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:57.238832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:57.238842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:57.238904 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:57.239049 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:57.239507 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:57.240273 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:57.240481 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:57.240564 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:57.241215 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:57.241216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:57.241216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:57.241220 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:57.241232 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:57.241236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:57.241279 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:57.241342 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 15:31:57.241365 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:31:57.246507 27744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27309e75f1934eec9e8fbf4011276a14: Generated new TSK 0
I20260924 15:31:57.246667 27744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:57.255929 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27663
2026-09-24T15:31:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:57 dist-test-slave-4g6f krb5kdc[27765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:57 dist-test-slave-4g6f krb5kdc[27765](info): set up 2 sockets
Sep 24 15:31:57 dist-test-slave-4g6f krb5kdc[27765](info): commencing operation
krb5kdc: starting...
W20260924 15:31:58.284997 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:58 dist-test-slave-4g6f krb5kdc[27765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:58.400110 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42457
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:46261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:58.479457 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:58.479723 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:58.479772 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:58.480953 27778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:58.481024 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:58.481047 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:58.481076 27778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:58.481107 27778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:58.482576 27778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:46261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:58.482856 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:58.483089 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:58.487927 27784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:58.488050 27778 server_base.cc:1034] running on GCE node
W20260924 15:31:58.487890 27787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:58.487888 27785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:58.488471 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:58.488754 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:58.489919 27778 hybrid_clock.cc:648] HybridClock initialized: now 1790263918489916 us; error 40 us; skew 500 ppm
Sep 24 15:31:58 dist-test-slave-4g6f krb5kdc[27765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263918, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:58.492271 27778 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:58.492717 27778 webserver.cc:460] Webserver started at http://127.5.98.254:37417/ using document root <none> and password file <none>
I20260924 15:31:58.492952 27778 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:58.493027 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:58.493173 27778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:58.494187 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "6a7c40207c124a2a8c15a193f6c21c7e"
format_stamp: "Formatted at 2026-09-24 15:31:58 on dist-test-slave-4g6f"
server_key: "ddf6a4501947f49b2ff3cf96ab91954d"
server_key_iv: "3b78987b89e89b957fc0fe342df0c55e"
server_key_version: "encryptionkey@0"
I20260924 15:31:58.494549 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "6a7c40207c124a2a8c15a193f6c21c7e"
format_stamp: "Formatted at 2026-09-24 15:31:58 on dist-test-slave-4g6f"
server_key: "ddf6a4501947f49b2ff3cf96ab91954d"
server_key_iv: "3b78987b89e89b957fc0fe342df0c55e"
server_key_version: "encryptionkey@0"
I20260924 15:31:58.495941 27778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:31:58.496680 27795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:58.496863 27778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:58.496974 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "6a7c40207c124a2a8c15a193f6c21c7e"
format_stamp: "Formatted at 2026-09-24 15:31:58 on dist-test-slave-4g6f"
server_key: "ddf6a4501947f49b2ff3cf96ab91954d"
server_key_iv: "3b78987b89e89b957fc0fe342df0c55e"
server_key_version: "encryptionkey@0"
I20260924 15:31:58.497051 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:58.500304 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:58.502116 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:58.502265 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:58.506225 27778 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:42457
I20260924 15:31:58.506255 27848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:42457 every 8 connection(s)
I20260924 15:31:58.506577 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:58.507539 27849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:58.509768 27849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e: Bootstrap starting.
I20260924 15:31:58.510443 27849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e: Neither blocks nor log segments found. Creating new log.
I20260924 15:31:58.510727 27849 log.cc:824] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:31:58.511415 27849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e: No bootstrap required, opened a new log
I20260924 15:31:58.513012 27849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } }
I20260924 15:31:58.513145 27849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:31:58.513167 27849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7c40207c124a2a8c15a193f6c21c7e, State: Initialized, Role: FOLLOWER
I20260924 15:31:58.513363 27849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } }
I20260924 15:31:58.513447 27849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:31:58.513499 27849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:31:58.513550 27849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:31:58.513990 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27778
I20260924 15:31:58.514117 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:58.514331 5515 external_mini_cluster.cc:1428] Setting key f7dc8e7a336ddeb105d9e5bc81bbbf67
I20260924 15:31:58.514655 27849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } }
I20260924 15:31:58.514810 27849 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7c40207c124a2a8c15a193f6c21c7e; no voters:
I20260924 15:31:58.514985 27849 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:31:58.515013 27854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:31:58.515199 27854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [term 1 LEADER]: Becoming Leader. State: Replica: 6a7c40207c124a2a8c15a193f6c21c7e, State: Running, Role: LEADER
I20260924 15:31:58.515249 27849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:31:58.515377 27854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } }
I20260924 15:31:58.515746 27854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7c40207c124a2a8c15a193f6c21c7e. Latest consensus state: current_term: 1 leader_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } } }
I20260924 15:31:58.515825 27854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:58.515987 27855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7c40207c124a2a8c15a193f6c21c7e" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42457 } } }
I20260924 15:31:58.516043 27855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e [sys.catalog]: This master's current role is: LEADER
I20260924 15:31:58.516752 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:58.514807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55137 (local address 127.5.98.254:42457)
0924 15:31:58.515013 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:58.515023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:58.515070 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:31:58.515199 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:58.515207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:58.515240 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:58.515700 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:58.516214 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:58.516217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:58.516217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:58.516220 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:58.516245 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:58.516255 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:58.516339 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:58.516493 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:31:58.516539 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260924 15:31:58.516743 27861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:31:58.517416 27861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:31:58.517822 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:31:58.518957 27861 catalog_manager.cc:1347] Generated new cluster ID: 598c25d4ed8a4775b0cbbfe4dcb1020c
I20260924 15:31:58.519007 27861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:31:58.520527 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:58.519355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36371 (local address 127.5.98.254:42457)
0924 15:31:58.519389 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:58.519391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:58.519605 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:31:58.519683 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:58.519684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:58.519690 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:58.519749 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:58.520321 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:58.520322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:58.520323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:31:58.520327 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:58.520341 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:31:58.520345 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:58.520390 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:58.520458 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:31:58.520471 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:31:58.546206 27861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:31:58.546373 27861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:31:58.553727 27861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7c40207c124a2a8c15a193f6c21c7e: Generated new TSK 0
I20260924 15:31:58.553911 27861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:31:58.560559 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27778
2026-09-24T15:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1304 ms)
[----------] 18 tests from AuthTokenIssuingTest (23645 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:31:58 dist-test-slave-4g6f krb5kdc[27881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:31:58 dist-test-slave-4g6f krb5kdc[27881](info): set up 2 sockets
Sep 24 15:31:58 dist-test-slave-4g6f krb5kdc[27881](info): commencing operation
krb5kdc: starting...
W20260924 15:31:59.596946 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:31:59 dist-test-slave-4g6f krb5kdc[27881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:31:59.714833 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42845
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:31:59.796658 27894 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:31:59.796900 27894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:59.796931 27894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:59.796974 27894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:59.798074 27894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:59.798141 27894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:59.798167 27894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:59.798197 27894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:59.798228 27894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:59.799706 27894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:59.799973 27894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:59.800192 27894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:31:59.805022 27894 server_base.cc:1034] running on GCE node
W20260924 15:31:59.805023 27899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:59.805043 27900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:59.805045 27902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:59.805502 27894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:59.805730 27894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:59.806900 27894 hybrid_clock.cc:648] HybridClock initialized: now 1790263919806888 us; error 29 us; skew 500 ppm
Sep 24 15:31:59 dist-test-slave-4g6f krb5kdc[27881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:59.812922 27894 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:31:59.813328 27894 webserver.cc:460] Webserver started at http://127.5.98.254:44441/ using document root <none> and password file <none>
I20260924 15:31:59.813548 27894 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:59.813614 27894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:59.813764 27894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:59.814798 27894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "48f960f4190d43bc8c22ebea840fe7d6"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "6a8b7809cdb1faecf3121b111edb216a"
server_key_iv: "fc81e63fbfabc2c8ef3d22267d0d3464"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.815186 27894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "48f960f4190d43bc8c22ebea840fe7d6"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "6a8b7809cdb1faecf3121b111edb216a"
server_key_iv: "fc81e63fbfabc2c8ef3d22267d0d3464"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.816573 27894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:31:59.817283 27909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:59.817474 27894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:31:59.817575 27894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "48f960f4190d43bc8c22ebea840fe7d6"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "6a8b7809cdb1faecf3121b111edb216a"
server_key_iv: "fc81e63fbfabc2c8ef3d22267d0d3464"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.817649 27894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:59.836550 27894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:31:59.838272 27894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:31:59.838428 27894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:31:59.842415 27894 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:42845
I20260924 15:31:59.842483 27963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:42845 every 8 connection(s)
I20260924 15:31:59.842824 27894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:31:59.843824 27964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:31:59.845324 27964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:31:59.849519 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27894
I20260924 15:31:59.849646 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:31:59.849804 5515 external_mini_cluster.cc:1428] Setting key 40a15223e79bd0c6d938313b34f10b40
Sep 24 15:31:59 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:31:59.855064 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.845955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42845 (local address 127.0.0.1:41612)
0924 15:31:59.846096 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.846138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:31:59.846601 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:31:59.846852 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:31:59.846854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:31:59.847104 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:31:59.847249 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:59.847253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:59.848294 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:59.848295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:59.848934 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:31:59.848936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:31:59.849429 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:31:59.849429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:31:59.849480 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:59.849976 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:31:59.849986 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:31:59.851755 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:31:59.853606 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:59.853612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:59.853614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:59.853801 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:59.854049 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:31:59.854050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:31:59.854050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:31:59.854141 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:31:59.854482 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:31:59.854487 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:31:59.854547 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 15:31:59.854653 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:31:59.854883 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":91,"thread_start_us":68,"threads_started":1}
I20260924 15:31:59.855063 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.846015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.5.98.254:42845)
0924 15:31:59.846182 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 15:31:59.846187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:31:59.846213 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:31:59.846725 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:31:59.846727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:31:59.846743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:31:59.846815 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:31:59.847327 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:59.848218 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:59.849006 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:31:59.849293 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:31:59.849448 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:31:59.851848 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:31:59.851855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:31:59.851855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:31:59.851861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:31:59.853514 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:59.853898 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:59.853900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:59.853901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:59.853980 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:31:59.854207 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:31:59.854208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:31:59.854208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:31:59.854409 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:31:59.854500 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:31:59.854686 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 15:31:59.854814 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260924 15:31:59.855822 27964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:31:59.856151 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.856044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44076)
0924 15:31:59.856092 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.856131 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:31:59.856312 27912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
I20260924 15:31:59.856760 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.856655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44086)
0924 15:31:59.856701 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.856723 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:31:59.856882 27964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:44431: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.98.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
I20260924 15:31:59.874118 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:44431
--webserver_interface=127.5.98.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:31:59.906091 27964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } attempt: 1
I20260924 15:31:59.906616 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.906485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44090)
0924 15:31:59.906548 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.906586 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:31:59.906984 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.906900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44092)
0924 15:31:59.906942 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.906966 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:31:59.907105 27964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:44431: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
W20260924 15:31:59.954330 27971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:31:59.954532 27971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:31:59.954550 27971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:31:59.954568 27971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:31:59.955708 27971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:31:59.955758 27971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:31:59.955773 27971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:31:59.955786 27971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:31:59.955798 27971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:31:59.957185 27971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.98.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:31:59.957361 27971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:31:59.957542 27971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:31:59.962412 27976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:59.962438 27971 server_base.cc:1034] running on GCE node
W20260924 15:31:59.962436 27977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:31:59.962421 27979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:31:59.962831 27971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:31:59.963070 27971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:31:59.964241 27971 hybrid_clock.cc:648] HybridClock initialized: now 1790263919964235 us; error 41 us; skew 500 ppm
Sep 24 15:31:59 dist-test-slave-4g6f krb5kdc[27881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:31:59.966596 27971 init.cc:377] Logged in from keytab as kudu/127.5.98.253@KRBTEST.COM (short username kudu)
I20260924 15:31:59.967029 27971 webserver.cc:460] Webserver started at http://127.5.98.253:46813/ using document root <none> and password file <none>
I20260924 15:31:59.967269 27971 fs_manager.cc:359] Metadata directory not provided
I20260924 15:31:59.967315 27971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:31:59.967444 27971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:31:59.968535 27971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data/instance:
uuid: "0e97f8b373744e839ee47b7ff88b1030"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "b1727094b70501bbd077000b10135aeb"
server_key_iv: "c9b7a3f3834ab002b2f22bf910a9bd2c"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.968901 27971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal/instance:
uuid: "0e97f8b373744e839ee47b7ff88b1030"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "b1727094b70501bbd077000b10135aeb"
server_key_iv: "c9b7a3f3834ab002b2f22bf910a9bd2c"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.970167 27971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:31:59.970893 27986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:31:59.971066 27971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:31:59.971163 27971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "0e97f8b373744e839ee47b7ff88b1030"
format_stamp: "Formatted at 2026-09-24 15:31:59 on dist-test-slave-4g6f"
server_key: "b1727094b70501bbd077000b10135aeb"
server_key_iv: "c9b7a3f3834ab002b2f22bf910a9bd2c"
server_key_version: "encryptionkey@0"
I20260924 15:31:59.971243 27971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:31:59.975266 27964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } attempt: 2
I20260924 15:31:59.975673 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.975536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44096)
0924 15:31:59.975592 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.975628 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:31:59.976001 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:31:59.975904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44100)
0924 15:31:59.975950 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:31:59.975977 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:31:59.976115 27964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:44431: Network error: Client connection negotiation failed: client connection to 127.5.98.253:44431: connect: Connection refused (error 111)
I20260924 15:31:59.997876 27971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:32:00.000289 27971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:32:00.000453 27971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:32:00.004730 27971 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.253:44431
I20260924 15:32:00.004863 28038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.253:44431 every 8 connection(s)
I20260924 15:32:00.005174 27971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
I20260924 15:32:00.006244 28039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:32:00.007793 28039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:00.008339 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 27971
I20260924 15:32:00.008473 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-1/wal/instance
I20260924 15:32:00.008687 5515 external_mini_cluster.cc:1428] Setting key 9b585abe9d2f2b91fa5d2a213a3970c1
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.252@KRBTEST.COM; defaulting to no policy
I20260924 15:32:00.018090 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.008553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.5.98.254:42845)
0924 15:32:00.008608 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.008611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.008872 (+ 261us) server_negotiation.cc:407] Connection header received
0924 15:32:00.009408 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.009410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.009429 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.009508 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.010135 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.010647 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.012192 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.012500 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.012565 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.015137 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.015139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.015139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.015150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.016746 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.017145 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.017146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.017147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.017209 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.017521 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.017522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.017522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.017643 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.017705 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.017953 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.018024 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:32:00.018244 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.008459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42845 (local address 127.0.0.1:41616)
0924 15:32:00.008722 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.008753 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.009279 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.009579 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.009582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.009824 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.010028 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.010032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.010755 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.010757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.012088 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.012091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.012613 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.012615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.012791 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.013330 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.013341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.015017 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.016828 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.016835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.016838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.017066 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.017287 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.017288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.017289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.017447 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.017710 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.017715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.017835 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.017924 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.018044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260924 15:32:00.018791 28039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.252@KRBTEST.COM" created.
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:00.026296 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.019180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.5.98.253:44431)
0924 15:32:00.019332 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.019337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.019359 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:32:00.019383 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.019386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.019401 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.019480 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.019824 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.020593 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.021345 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.021742 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.021814 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.023542 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.023549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.023549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.023555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.025151 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.025549 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.025551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.025552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.025603 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.025809 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.025810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.025810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.025938 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.026006 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.026157 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.026222 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 15:32:00.026424 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.019122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44116)
0924 15:32:00.019171 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.019182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.019224 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.019511 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.019512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.019692 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.019751 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.019753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.020684 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.020685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.021232 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.021234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.021810 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.021811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.021881 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.022283 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.022288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.023446 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.025239 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.025240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.025240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.025463 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.025672 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.025673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.025674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.025743 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.026005 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.026006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.026054 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.026136 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.026282 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":50}
I20260924 15:32:00.027077 28039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:00.027442 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.027304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35904)
0924 15:32:00.027358 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.027414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:32:00.027590 27988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
I20260924 15:32:00.028002 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.027939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35914)
0924 15:32:00.027961 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.027985 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 15:32:00.028126 28039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:37957: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
I20260924 15:32:00.034662 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:37957
--webserver_interface=127.5.98.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:32:00.067358 27964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } attempt: 3
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:00.074728 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.067826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44118)
0924 15:32:00.067879 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.067895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.067977 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.068151 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.068152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.068315 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.068379 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.068381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.069062 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.069063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.069599 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.069601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.070005 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.070007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.070097 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.070620 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.070625 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.072018 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.073623 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.073624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.073624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.073846 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.074057 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.074058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.074058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.074124 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.074384 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.074386 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.074432 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.074524 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.074627 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 15:32:00.074729 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.067909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.5.98.253:44431)
0924 15:32:00.067985 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.067988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.068001 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:32:00.068044 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.068045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.068055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.068125 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.068482 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.068971 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.069693 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.069926 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.070005 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.072110 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.072111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.072111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.072119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.073534 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.073913 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.073915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.073915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.073982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.074194 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.074195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.074195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.074308 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.074380 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.074550 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.074628 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:32:00.075515 27964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:00.075872 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.075769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35922)
0924 15:32:00.075819 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.075847 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:32:00.076231 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.076147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35926)
0924 15:32:00.076192 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.076215 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:32:00.076329 27964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:37957: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
I20260924 15:32:00.077334 28039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } attempt: 1
I20260924 15:32:00.077688 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.077554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35942)
0924 15:32:00.077617 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.077654 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:32:00.078006 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.077938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35944)
0924 15:32:00.077962 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.077987 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 15:32:00.078100 28039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:37957: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
I20260924 15:32:00.107470 27964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } attempt: 1
I20260924 15:32:00.108003 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.107846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35960)
0924 15:32:00.107932 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.107975 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 15:32:00.108448 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.108360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35970)
0924 15:32:00.108406 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.108432 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:32:00.108583 27964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:37957: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
W20260924 15:32:00.116871 28046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:32:00.117063 28046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:32:00.117079 28046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:32:00.117098 28046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:32:00.118122 28046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:32:00.118155 28046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:32:00.118168 28046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:32:00.118180 28046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:32:00.118192 28046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:32:00.119746 28046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:42845,127.5.98.253:44431,127.5.98.252:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.98.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:32:00.119933 28046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:32:00.120097 28046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:32:00.124938 28052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:00.124908 28055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:00.124918 28053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:32:00.125114 28046 server_base.cc:1034] running on GCE node
I20260924 15:32:00.125398 28046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:32:00.125633 28046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:32:00.126796 28046 hybrid_clock.cc:648] HybridClock initialized: now 1790263920126784 us; error 30 us; skew 500 ppm
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263920, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:32:00.129266 28046 init.cc:377] Logged in from keytab as kudu/127.5.98.252@KRBTEST.COM (short username kudu)
I20260924 15:32:00.129727 28046 webserver.cc:460] Webserver started at http://127.5.98.252:40177/ using document root <none> and password file <none>
I20260924 15:32:00.129971 28046 fs_manager.cc:359] Metadata directory not provided
I20260924 15:32:00.130038 28046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:32:00.130191 28046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:32:00.131207 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data/instance:
uuid: "a1669fded45d4f5ea48bb60d1850543b"
format_stamp: "Formatted at 2026-09-24 15:32:00 on dist-test-slave-4g6f"
server_key: "075762bd43baa3341d9c1cc1a494b256"
server_key_iv: "9085d813dd926ce2b708abf384344133"
server_key_version: "encryptionkey@0"
I20260924 15:32:00.131579 28046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal/instance:
uuid: "a1669fded45d4f5ea48bb60d1850543b"
format_stamp: "Formatted at 2026-09-24 15:32:00 on dist-test-slave-4g6f"
server_key: "075762bd43baa3341d9c1cc1a494b256"
server_key_iv: "9085d813dd926ce2b708abf384344133"
server_key_version: "encryptionkey@0"
I20260924 15:32:00.132900 28046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:32:00.133636 28063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:32:00.133808 28046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:32:00.133903 28046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "a1669fded45d4f5ea48bb60d1850543b"
format_stamp: "Formatted at 2026-09-24 15:32:00 on dist-test-slave-4g6f"
server_key: "075762bd43baa3341d9c1cc1a494b256"
server_key_iv: "9085d813dd926ce2b708abf384344133"
server_key_version: "encryptionkey@0"
I20260924 15:32:00.133991 28046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:32:00.145592 28046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:32:00.146281 28039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } attempt: 2
I20260924 15:32:00.146723 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.146559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35972)
0924 15:32:00.146626 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.146686 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:32:00.147130 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.147012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:35980)
0924 15:32:00.147064 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.147095 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:32:00.147468 28039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:37957: Network error: Client connection negotiation failed: client connection to 127.5.98.252:37957: connect: Connection refused (error 111)
I20260924 15:32:00.147965 28046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:32:00.148108 28046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:32:00.152236 28046 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.252:37957
I20260924 15:32:00.152307 28116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.252:37957 every 8 connection(s)
I20260924 15:32:00.152640 28046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
I20260924 15:32:00.153671 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:32:00.155169 28117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:00.159567 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 28046
I20260924 15:32:00.159703 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263655576751-5515-0/minicluster-data/master-2/wal/instance
I20260924 15:32:00.159919 5515 external_mini_cluster.cc:1428] Setting key 2d7d48976990891e37b636eb8ebe987c
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263920, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:32:00.164495 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.155827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.5.98.254:42845)
0924 15:32:00.155896 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.155898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.156090 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:32:00.156563 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.156565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.156576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.156635 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.157199 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.157663 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.158570 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.158746 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.158791 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.161084 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.161086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.161086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.161093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.163075 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.163500 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.163501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.163502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.163562 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.163843 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.163844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.163845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.163959 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.164039 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.164311 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.164397 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:32:00.164644 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.155819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:42845 (local address 127.0.0.1:41630)
0924 15:32:00.155967 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.156007 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.156441 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.156713 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.156716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.156939 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.157122 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.157126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.157741 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.157742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.158500 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.158502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.158814 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.158815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.158926 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.159306 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.159314 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.160988 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.163185 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.163193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.163195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.163416 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.163632 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.163633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.163634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.163770 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.164032 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.164039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.164175 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.164277 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.164421 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:00.165134 28117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:00.168550 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.160314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.5.98.252:37957)
0924 15:32:00.160474 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.160478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.160496 (+ 18us) server_negotiation.cc:407] Connection header received
0924 15:32:00.160562 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.160564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.160584 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.160653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.161274 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.161931 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.162715 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.163024 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.163104 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.165580 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.165587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.165587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.165594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.167202 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.167627 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.167630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.167630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.167715 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.167932 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.167933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.167933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.168065 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.168215 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.168306 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.168423 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263920, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:00.172129 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.165397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:44431 (local address 127.0.0.1:44132)
0924 15:32:00.165517 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.165528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.165575 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.165721 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.165722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.165865 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.165940 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.165942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.166849 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.166850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.167377 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.167379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.167795 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.167796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.167871 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.168268 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.168273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.169520 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.171076 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.171077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.171077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.171241 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.171452 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.171453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.171453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.171521 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.171828 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.171829 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.171868 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.171957 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.172042 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 15:32:00.172143 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.165467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.5.98.253:44431)
0924 15:32:00.165517 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.165520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.165561 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:32:00.165640 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.165641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.165651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.165720 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.166020 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.166757 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.167449 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.167733 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.167794 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.169606 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.169607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.169608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.169615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.170992 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.171324 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.171325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.171325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.171382 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.171594 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.171594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.171595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.171761 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.171833 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.171990 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.172059 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:32:00.172513 28117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263920, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:00.176858 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.169286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.5.98.254:42845)
0924 15:32:00.169325 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.169326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.169334 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:00.169354 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.169355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.169363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.169433 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.169891 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.170616 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.171259 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.171447 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.171520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.174132 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.174135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.174136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.174146 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.175795 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.176157 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.176158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.176158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.176214 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.176363 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.176364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.176364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.176480 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.176650 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.176690 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.176765 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:32:00.179527 28119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.172802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:36006)
0924 15:32:00.172832 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.172842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.172884 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.173021 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.173021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.173132 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.173157 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.173158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.173991 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.173992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.174701 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.174703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.175123 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.175124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.175172 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.175534 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.175538 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.176841 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.178581 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.178582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.178583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.178786 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.178965 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.178966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.178966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.179028 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.179241 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.179242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.179280 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.179349 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.179455 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 15:32:00.179646 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.172850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.5.98.252:37957)
0924 15:32:00.172882 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.172884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.172892 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:00.172945 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.172946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.172954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.173016 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.173230 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.173926 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.174792 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.175053 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.175118 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.176947 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.176948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.176949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.176956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.178522 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.178858 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.178859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.178859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.178905 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.179086 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.179087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.179087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.179171 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.179229 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.179364 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.179541 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":14}
I20260924 15:32:00.180950 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: Bootstrap starting.
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:00.181810 28117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:00.182132 28117 log.cc:824] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:32:00.182976 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: No bootstrap required, opened a new log
I20260924 15:32:00.183743 27964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } attempt: 2
I20260924 15:32:00.184621 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.177469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.5.98.253:44431)
0924 15:32:00.177507 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.177509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.177517 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:00.177560 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.177561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.177571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.177638 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.178066 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.178485 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.179426 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.179870 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.179941 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.181936 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.181937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.181937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.181943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.183404 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.183782 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.183783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.183784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.183857 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.184081 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.184082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.184082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.184224 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.184367 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.184471 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.184537 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:00.185022 28117 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.185187 28117 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1669fded45d4f5ea48bb60d1850543b, State: Initialized, Role: FOLLOWER
I20260924 15:32:00.185279 28117 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.185583 28117 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:00.185580 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.185680 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.186383 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 15:32:00.187989 28135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:32:00.188040 28135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:00.187634 27936 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "48f960f4190d43bc8c22ebea840fe7d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41638
I20260924 15:32:00.187731 28046 master_runner.cc:186] Error getting master registration for 127.5.98.254:42845: OK, instance_id { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" instance_seqno: 1790263919841259 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:32:00.190562 28046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:32:00.191043 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.183999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.5.98.252:37957)
0924 15:32:00.184034 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.184036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.184043 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:32:00.184119 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.184120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.184128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.184179 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.184794 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.185468 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.186224 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.186481 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.186742 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.188482 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.188484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.188484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.188491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.189842 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.190208 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.190209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.190210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.190267 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.190488 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.190489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.190489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.190647 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.190711 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.190891 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.190959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:32:00.191056 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.183964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:36008)
0924 15:32:00.184005 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.184015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.184049 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.184205 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.184206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.184353 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.184392 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.184394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.185606 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.185607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.186140 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.186142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.186564 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.186565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.186615 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.186997 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.187003 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.188377 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.189922 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.189923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.189923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.190132 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.190339 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.190339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.190340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.190406 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.190707 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.190708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.190762 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.190850 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.190959 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 15:32:00.192368 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: Bootstrap starting.
I20260924 15:32:00.192875 27964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:00.193113 27964 log.cc:824] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:32:00.193790 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: No bootstrap required, opened a new log
I20260924 15:32:00.194669 27964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.194761 27964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f960f4190d43bc8c22ebea840fe7d6, State: Initialized, Role: FOLLOWER
I20260924 15:32:00.194841 27964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.195137 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.195180 27964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:00.195225 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.195226 27936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:32:00.195385 27936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:00.196184 27936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.196827 27936 leader_election.cc:288] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [CANDIDATE]: Term 1 election: Requested vote from peers 0e97f8b373744e839ee47b7ff88b1030 (127.5.98.253:44431), a1669fded45d4f5ea48bb60d1850543b (127.5.98.252:37957)
I20260924 15:32:00.197535 28091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "48f960f4190d43bc8c22ebea840fe7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a1669fded45d4f5ea48bb60d1850543b"
I20260924 15:32:00.197674 28091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:00.198544 28091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f960f4190d43bc8c22ebea840fe7d6 in term 1.
W20260924 15:32:00.198798 28151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:32:00.198854 28151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:32:00.197980 27894 master_runner.cc:186] Error getting master registration for 127.5.98.253:44431: OK, instance_id { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" instance_seqno: 1790263920003363 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:32:00.200502 27894 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:32:00.200609 27911 leader_election.cc:302] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f960f4190d43bc8c22ebea840fe7d6, a1669fded45d4f5ea48bb60d1850543b; no voters:
I20260924 15:32:00.200716 28138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:32:00.200855 28138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [term 1 LEADER]: Becoming Leader. State: Replica: 48f960f4190d43bc8c22ebea840fe7d6, State: Running, Role: LEADER
I20260924 15:32:00.200927 28138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.201253 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f960f4190d43bc8c22ebea840fe7d6. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.201342 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:32:00.201570 28162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:32:00.201736 28162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:32:00.203562 28091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [term 1 FOLLOWER]: Refusing update from remote peer 48f960f4190d43bc8c22ebea840fe7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:32:00.203878 28138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:32:00.205133 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f960f4190d43bc8c22ebea840fe7d6. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.205226 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.208537 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.198890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.5.98.254:42845)
0924 15:32:00.198935 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.198937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.199487 (+ 550us) server_negotiation.cc:407] Connection header received
0924 15:32:00.199584 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.199585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.199594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.199621 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.200365 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.201095 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.202472 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.202752 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.202792 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.205092 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.205093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.205094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.205100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.206700 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.207291 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.207292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.207293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.207372 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.207763 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.207765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.207765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.207902 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.207997 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.208362 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.208439 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:00.212237 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.198346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.5.98.253:44431)
0924 15:32:00.198379 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.198380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.199257 (+ 877us) server_negotiation.cc:407] Connection header received
0924 15:32:00.199383 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.199384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.199395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.199436 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.200646 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.201300 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.202828 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.203399 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.203443 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.208813 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.208814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.208815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.208822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.210600 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.211175 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.211176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.211176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.211252 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.211538 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.211539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.211539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.211680 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.211748 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.212060 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.212133 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:32:00.215224 28163 mvcc.cc:205] Tried to move back new op lower bound from 7332921017151500288 to 7332921017143558144. Current Snapshot: MvccSnapshot[applied={T|T < 7332921017151500288}]
I20260924 15:32:00.215670 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.215759 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:32:00.216241 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.216326 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.216588 28164 mvcc.cc:205] Tried to move back new op lower bound from 7332921017151500288 to 7332921017143558144. Current Snapshot: MvccSnapshot[applied={T|T < 7332921017151500288}]
I20260924 15:32:00.217991 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.199247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.5.98.252:37957)
0924 15:32:00.199280 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.199282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.208938 (+ 9656us) server_negotiation.cc:407] Connection header received
0924 15:32:00.209089 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.209102 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.209111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.209138 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.210072 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.210821 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.212298 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.212595 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.212643 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.214896 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.214908 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.214908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.214915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.216775 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.217173 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.217174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.217174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.217223 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.217423 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.217424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.217424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.217561 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.217643 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.217808 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.217875 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:00.218521 28162 catalog_manager.cc:1347] Generated new cluster ID: bf240e8b53314f34825284326a916b22
I20260924 15:32:00.218575 28162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:32:00.232374 28162 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:32:00.233207 28162 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:32:00.238601 28039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } attempt: 3
I20260924 15:32:00.240361 28162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6: Generated new TSK 0
I20260924 15:32:00.240525 28162 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 24 15:32:00 dist-test-slave-4g6f krb5kdc[27881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263919, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:00.246321 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.238818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:37957 (local address 127.0.0.1:36034)
0924 15:32:00.238901 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:00.238911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:00.238956 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:00.239102 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:00.239103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:00.239262 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:00.239300 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.239302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.240176 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.240177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.240846 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:00.240849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.241307 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.241308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:00.241362 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.241769 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:00.241773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:00.243167 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:00.245177 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.245178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.245179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.245404 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.245638 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:00.245638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:00.245639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:00.245701 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:00.245973 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:00.245974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:00.246016 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:32:00.246076 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:32:00.246174 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 15:32:00.246646 28121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:00.238904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.5.98.252:37957)
0924 15:32:00.238952 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.238954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.238963 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:32:00.239001 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.239002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.239011 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.239062 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.239388 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.240087 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.240935 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:00.241218 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:00.241294 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:00.243260 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:00.243262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:00.243262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:00.243269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:00.245090 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.245514 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.245515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.245516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.245568 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:00.245765 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:00.245766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:00.245766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:00.245893 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:00.245947 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:00.246437 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 15:32:00.246526 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:32:00.248450 28039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: Bootstrap starting.
I20260924 15:32:00.249152 28039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:00.249447 28039 log.cc:824] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: Log is configured to *not* fsync() on all Append() calls
I20260924 15:32:00.253319 28039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: No bootstrap required, opened a new log
I20260924 15:32:00.255446 28039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.255577 28039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e97f8b373744e839ee47b7ff88b1030, State: Initialized, Role: FOLLOWER
I20260924 15:32:00.255647 28039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } }
I20260924 15:32:00.255928 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.256143 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.256040 28013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "48f960f4190d43bc8c22ebea840fe7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e97f8b373744e839ee47b7ff88b1030"
I20260924 15:32:00.256017 28012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:00.256956 28012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [term 1 FOLLOWER]: Refusing update from remote peer 48f960f4190d43bc8c22ebea840fe7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:32:00.257258 28138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:32:00.257854 28013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f960f4190d43bc8c22ebea840fe7d6 in term 1.
I20260924 15:32:00.255959 28039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:00.258919 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f960f4190d43bc8c22ebea840fe7d6. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.259035 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:00.259080 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.259140 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:32:00.260434 28187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:32:00.260506 28187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:32:00.268229 28174 mvcc.cc:205] Tried to move back new op lower bound from 7332921017301401600 to 7332921017143558144. Current Snapshot: MvccSnapshot[applied={T|T < 7332921017151500288}]
W20260924 15:32:00.273138 27967 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.271325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.5.98.254:42845)
0924 15:32:00.271366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.271368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.272226 (+ 858us) server_negotiation.cc:407] Connection header received
0924 15:32:00.272328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.272329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.272340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.272371 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.272963 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41664: BlockingRecv error: recv got EOF from 127.0.0.1:41664 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 15:32:00.273984 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.271531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.5.98.253:44431)
0924 15:32:00.271567 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.271571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.273468 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 15:32:00.273564 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.273564 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.273574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.273618 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.273922 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44162: BlockingRecv error: recv got EOF from 127.0.0.1:44162 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:32:00.274760 28121 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.271891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.5.98.252:37957)
0924 15:32:00.271933 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.271935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.274294 (+ 2359us) server_negotiation.cc:407] Connection header received
0924 15:32:00.274389 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.274390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.274400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.274427 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.274703 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36038: BlockingRecv error: recv got EOF from 127.0.0.1:36038 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:32:00.277084 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.276448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.5.98.253:44431)
0924 15:32:00.276485 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.276487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.276607 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:32:00.276702 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.276703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.276718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.276745 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.277046 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44168: BlockingRecv error: recv got EOF from 127.0.0.1:44168 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:32:00.280051 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.279318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.5.98.253:44431)
0924 15:32:00.279348 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.279349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.279562 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:32:00.279647 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.279648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.279677 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.279705 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.280013 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44180: BlockingRecv error: recv got EOF from 127.0.0.1:44180 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 15:32:00.283939 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.283180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.5.98.253:44431)
0924 15:32:00.283211 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.283213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.283427 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:32:00.283517 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.283518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.283527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.283555 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.283900 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44196: BlockingRecv error: recv got EOF from 127.0.0.1:44196 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:00.289696 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.289001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.5.98.253:44431)
0924 15:32:00.289033 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.289034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.289238 (+ 204us) server_negotiation.cc:407] Connection header received
0924 15:32:00.289329 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.289329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.289339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.289366 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.289656 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44198: BlockingRecv error: recv got EOF from 127.0.0.1:44198 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:00.299537 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.298773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.5.98.253:44431)
0924 15:32:00.298804 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.298805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.299053 (+ 248us) server_negotiation.cc:407] Connection header received
0924 15:32:00.299146 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.299147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.299156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.299184 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.299485 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44212: BlockingRecv error: recv got EOF from 127.0.0.1:44212 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 15:32:00.317481 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.316640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44224 (local address 127.5.98.253:44431)
0924 15:32:00.316674 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.316676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.316910 (+ 234us) server_negotiation.cc:407] Connection header received
0924 15:32:00.317031 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.317032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.317042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.317093 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.317434 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44224: BlockingRecv error: recv got EOF from 127.0.0.1:44224 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:32:00.351802 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.350796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.5.98.253:44431)
0924 15:32:00.350837 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.350839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.351164 (+ 325us) server_negotiation.cc:407] Connection header received
0924 15:32:00.351272 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.351273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.351301 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.351331 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.351743 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44234: BlockingRecv error: recv got EOF from 127.0.0.1:44234 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:32:00.418004 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.417273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.5.98.253:44431)
0924 15:32:00.417389 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.417391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.417479 (+ 88us) server_negotiation.cc:407] Connection header received
0924 15:32:00.417535 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.417536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.417546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.417599 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.417935 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44244: BlockingRecv error: recv got EOF from 127.0.0.1:44244 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 15:32:00.548184 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.547249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.5.98.253:44431)
0924 15:32:00.547375 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.547378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.547530 (+ 152us) server_negotiation.cc:407] Connection header received
0924 15:32:00.547647 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.547648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.547687 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.547752 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.548117 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44260: BlockingRecv error: recv got EOF from 127.0.0.1:44260 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 15:32:00.806070 28043 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:00.805370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.5.98.253:44431)
0924 15:32:00.805456 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:00.805458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:00.805487 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:32:00.805574 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:00.805575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:00.805585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:00.805654 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:00.806003 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44274: BlockingRecv error: recv got EOF from 127.0.0.1:44274 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:32:00.810222 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48f960f4190d43bc8c22ebea840fe7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f960f4190d43bc8c22ebea840fe7d6" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 42845 } } peers { permanent_uuid: "0e97f8b373744e839ee47b7ff88b1030" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 44431 } } peers { permanent_uuid: "a1669fded45d4f5ea48bb60d1850543b" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 37957 } } }
I20260924 15:32:00.810433 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f960f4190d43bc8c22ebea840fe7d6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:32:01.188509 28135 catalog_manager.cc:1259] Loaded cluster ID: bf240e8b53314f34825284326a916b22
I20260924 15:32:01.188596 28135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: loading cluster ID for follower catalog manager: success
I20260924 15:32:01.189483 28135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: acquiring CA information for follower catalog manager: success
I20260924 15:32:01.189803 28135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1669fded45d4f5ea48bb60d1850543b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:32:01.260914 28187 catalog_manager.cc:1259] Loaded cluster ID: bf240e8b53314f34825284326a916b22
I20260924 15:32:01.260989 28187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: loading cluster ID for follower catalog manager: success
I20260924 15:32:01.261806 28187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: acquiring CA information for follower catalog manager: success
I20260924 15:32:01.262135 28187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e97f8b373744e839ee47b7ff88b1030: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:32:01.322016 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:01.319133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.5.98.253:44431)
0924 15:32:01.319250 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:01.319253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:01.319326 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:32:01.319443 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:01.319444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:01.319515 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:01.319623 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:01.319800 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.320351 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.321131 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.321395 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.321467 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:01.321648 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:01.321770 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:01.321859 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:01.321897 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:32:01.321918 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":50,"threads_started":1}
I20260924 15:32:01.326529 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:01.323830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.5.98.252:37957)
0924 15:32:01.323938 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:01.323940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:01.324049 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:32:01.324118 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:01.324118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:01.324150 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:01.324213 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:01.324345 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.324785 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.325504 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.325859 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.325941 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:01.326121 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:01.326245 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:01.326315 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:01.326426 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 15:32:01.326445 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260924 15:32:01.330983 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:01.328095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.5.98.254:42845)
0924 15:32:01.328198 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:01.328200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:01.328212 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:32:01.328259 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:01.328260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:01.328308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:01.328394 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:01.328486 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.329254 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330048 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.330284 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330360 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:01.330536 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:01.330639 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:01.330687 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:01.330851 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:32:01.330874 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260924 15:32:01.331189 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:01.328399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.5.98.252:37957)
0924 15:32:01.328441 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:01.328443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:01.328602 (+ 159us) server_negotiation.cc:407] Connection header received
0924 15:32:01.328674 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:01.328675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:01.328686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:01.328734 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:01.328881 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.329308 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330498 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.330699 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330768 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:01.330940 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:01.330970 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:01.331015 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:01.331105 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:32:01.331137 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:32:01.331259 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:01.328081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.5.98.253:44431)
0924 15:32:01.328118 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:01.328120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:01.328457 (+ 337us) server_negotiation.cc:407] Connection header received
0924 15:32:01.328523 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:01.328524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:01.328532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:01.328602 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:01.328715 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.329394 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330452 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:01.330744 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:01.330799 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:01.331007 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:01.331036 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:01.331085 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:01.331173 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 15:32:01.331195 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:01.332798 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27894
I20260924 15:32:01.339267 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 27971
I20260924 15:32:01.344015 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 28046
2026-09-24T15:32:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2784 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:32:01 dist-test-slave-4g6f krb5kdc[28274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:32:01 dist-test-slave-4g6f krb5kdc[28274](info): set up 2 sockets
Sep 24 15:32:01 dist-test-slave-4g6f krb5kdc[28274](info): commencing operation
krb5kdc: starting...
W20260924 15:32:02.436905 5515 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/test-user.keytab.
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.98.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
Entry for principal kudu/127.5.98.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab.
I20260924 15:32:02.550490 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44205
--webserver_interface=127.5.98.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:32:02.629834 28288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:32:02.630064 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:32:02.630093 28288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:32:02.630137 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:32:02.631258 28288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:32:02.631322 28288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:32:02.631346 28288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:32:02.631384 28288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:32:02.631412 28288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:32:02.632902 28288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.254.keytab
--principal=kudu/127.5.98.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.254:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.98.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:32:02.633185 28288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:32:02.633415 28288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:32:02.638341 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:02.638319 28294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:02.638353 28296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:32:02.638396 28288 server_base.cc:1034] running on GCE node
I20260924 15:32:02.638792 28288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:32:02.639091 28288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:32:02.640255 28288 hybrid_clock.cc:648] HybridClock initialized: now 1790263922640207 us; error 61 us; skew 500 ppm
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:32:02.642765 28288 init.cc:377] Logged in from keytab as kudu/127.5.98.254@KRBTEST.COM (short username kudu)
I20260924 15:32:02.643198 28288 webserver.cc:460] Webserver started at http://127.5.98.254:40001/ using document root <none> and password file <none>
I20260924 15:32:02.643440 28288 fs_manager.cc:359] Metadata directory not provided
I20260924 15:32:02.643486 28288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:32:02.643615 28288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:32:02.644655 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data/instance:
uuid: "18d3cb819099404f98b78a23dab52182"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "fc8f34cffc9009a6be16ddf31b64ae64"
server_key_iv: "d86c8f41b14eefb66a8d1dd603071989"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.644948 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance:
uuid: "18d3cb819099404f98b78a23dab52182"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "fc8f34cffc9009a6be16ddf31b64ae64"
server_key_iv: "d86c8f41b14eefb66a8d1dd603071989"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.646211 28288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:32:02.646953 28303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:32:02.647179 28288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:32:02.647271 28288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal
uuid: "18d3cb819099404f98b78a23dab52182"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "fc8f34cffc9009a6be16ddf31b64ae64"
server_key_iv: "d86c8f41b14eefb66a8d1dd603071989"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.647349 28288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:32:02.656767 28288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:32:02.658560 28288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:32:02.658669 28288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:32:02.662691 28288 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.254:44205
I20260924 15:32:02.662750 28356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.254:44205 every 8 connection(s)
I20260924 15:32:02.663128 28288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/data/info.pb
I20260924 15:32:02.664219 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:32:02.664477 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 28288
I20260924 15:32:02.664616 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-0/wal/instance
I20260924 15:32:02.664829 5515 external_mini_cluster.cc:1428] Setting key d6a51ee5d6ba238c943cf7d9314e844e
I20260924 15:32:02.665874 28357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.98.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.253@KRBTEST.COM" created.
I20260924 15:32:02.677094 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.666646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44205 (local address 127.0.0.1:35740)
0924 15:32:02.666808 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.666853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.667461 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.667781 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.667784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.668183 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.668451 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.668459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.669524 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.669525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.670865 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.670869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.671388 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.671390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.671484 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.671979 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.671991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.673608 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.675459 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.675466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.675468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.675782 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.676034 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.676035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.676035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.676127 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.676515 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.676520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.676584 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.676691 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.676920 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":94,"thread_start_us":68,"threads_started":1}
I20260924 15:32:02.677098 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.666803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.5.98.254:44205)
0924 15:32:02.667251 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.667255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.667290 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:32:02.667629 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.667632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.667651 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.667750 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.668588 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.669417 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.670965 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.671269 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.671350 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.673705 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.673712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.673713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.673726 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.675351 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.675876 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.675879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.675879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.675961 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.676195 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.676196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.676196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.676439 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.676534 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.676726 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.676841 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":42,"threads_started":1}
I20260924 15:32:02.677819 28357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:02.678150 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.678030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43240)
0924 15:32:02.678077 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.678129 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:32:02.678284 28307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
I20260924 15:32:02.678692 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.678594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43254)
0924 15:32:02.678645 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.678669 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:32:02.678797 28357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:34705: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
Entry for principal kudu/127.5.98.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab.
I20260924 15:32:02.692446 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:34705
--webserver_interface=127.5.98.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:32:02.728027 28357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } attempt: 1
I20260924 15:32:02.728499 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.728359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43256)
0924 15:32:02.728426 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.728465 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:32:02.728977 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.728877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43262)
0924 15:32:02.728926 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.728952 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:32:02.729094 28357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:34705: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
W20260924 15:32:02.773437 28364 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:32:02.773653 28364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:32:02.773681 28364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:32:02.773722 28364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:32:02.774822 28364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:32:02.774890 28364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:32:02.774916 28364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:32:02.774948 28364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:32:02.774976 28364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:32:02.776491 28364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.253.keytab
--principal=kudu/127.5.98.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.253:34705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.98.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:32:02.776759 28364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:32:02.776988 28364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:32:02.781662 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:02.781662 28370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:02.781706 28369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:32:02.781986 28364 server_base.cc:1034] running on GCE node
I20260924 15:32:02.782265 28364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:32:02.782610 28364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:32:02.783746 28364 hybrid_clock.cc:648] HybridClock initialized: now 1790263922783732 us; error 39 us; skew 500 ppm
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:32:02.786376 28364 init.cc:377] Logged in from keytab as kudu/127.5.98.253@KRBTEST.COM (short username kudu)
I20260924 15:32:02.786922 28364 webserver.cc:460] Webserver started at http://127.5.98.253:45919/ using document root <none> and password file <none>
I20260924 15:32:02.787158 28364 fs_manager.cc:359] Metadata directory not provided
I20260924 15:32:02.787204 28364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:32:02.787377 28364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:32:02.788558 28364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data/instance:
uuid: "830c9dd0aa3644e39fbba5b4318e0676"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "228fd67a2a6edd6d90e2101e1c4e30df"
server_key_iv: "499c451fd480c1d4fa939bb0afbc9dc0"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.788934 28364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal/instance:
uuid: "830c9dd0aa3644e39fbba5b4318e0676"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "228fd67a2a6edd6d90e2101e1c4e30df"
server_key_iv: "499c451fd480c1d4fa939bb0afbc9dc0"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.790338 28364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:32:02.791090 28379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:32:02.791275 28364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:32:02.791414 28364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal
uuid: "830c9dd0aa3644e39fbba5b4318e0676"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "228fd67a2a6edd6d90e2101e1c4e30df"
server_key_iv: "499c451fd480c1d4fa939bb0afbc9dc0"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.791507 28364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:32:02.797232 28357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } attempt: 2
I20260924 15:32:02.797585 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.797468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43264)
0924 15:32:02.797526 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.797559 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:32:02.797904 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.797811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43272)
0924 15:32:02.797855 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.797879 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:32:02.797999 28357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.253:34705: Network error: Client connection negotiation failed: client connection to 127.5.98.253:34705: connect: Connection refused (error 111)
I20260924 15:32:02.803963 28364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:32:02.805666 28364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:32:02.805823 28364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:32:02.809876 28364 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.253:34705
I20260924 15:32:02.809908 28433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.253:34705 every 8 connection(s)
I20260924 15:32:02.810251 28364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/data/info.pb
I20260924 15:32:02.811174 28434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:32:02.812664 28434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:02.816825 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 28364
I20260924 15:32:02.816946 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-1/wal/instance
I20260924 15:32:02.817173 5515 external_mini_cluster.cc:1428] Setting key 08a5fc500044f747bac83a3436641af5
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:32:02.821571 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.813225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44205 (local address 127.0.0.1:35748)
0924 15:32:02.813385 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.813426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.813856 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.814083 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.814086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.814303 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.814546 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.814550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.815182 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.815183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.816039 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.816042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.816401 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.816402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.816502 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.817047 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.817064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.818681 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.820343 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.820349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.820350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.820554 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.820757 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.820758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.820758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.820846 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.821118 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.821122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.821222 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.821299 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.821429 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260924 15:32:02.821585 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.813275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.5.98.254:44205)
0924 15:32:02.813336 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.813338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.813506 (+ 168us) server_negotiation.cc:407] Connection header received
0924 15:32:02.813934 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.813935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.813945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.814010 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.814619 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.815107 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.816108 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.816298 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.816352 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.818776 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.818778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.818778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.818786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.820273 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.820625 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.820626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.820627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.820689 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.820910 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.820911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.820911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.821056 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.821124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.821345 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.821436 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:32:02.822497 28434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.98.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.98.252@KRBTEST.COM" created.
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:02.833519 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.822734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43278)
0924 15:32:02.822783 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.822793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.822833 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.823143 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.823144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.823252 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.823297 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.823298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.824678 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.824679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.825165 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.825166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.825852 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.825853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.825907 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.826354 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.826358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.830424 (+ 4066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.832310 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.832311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.832312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.832543 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.832748 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.832749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.832750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.832820 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.833084 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.833085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.833135 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.833211 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.833333 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:32:02.833534 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.822790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.5.98.253:34705)
0924 15:32:02.822951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.822957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.822983 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:32:02.823010 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.823012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.823029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.823126 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.823371 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.824569 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.825252 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.825765 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.825857 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.830523 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.830531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.830532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.830539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.832229 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.832627 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.832630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.832630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.832682 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.832883 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.832884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.832884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.833012 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.833081 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.833253 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.833410 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260924 15:32:02.834213 28434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:02.834554 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.834425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41262)
0924 15:32:02.834487 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.834533 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:32:02.834837 28383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
I20260924 15:32:02.835282 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.835194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41274)
0924 15:32:02.835241 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.835266 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:32:02.835410 28434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:33371: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
Entry for principal kudu/127.5.98.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab.
I20260924 15:32:02.858625 5515 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskC7bDtI/build/release/bin/kudu
/tmp/dist-test-taskC7bDtI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:33371
--webserver_interface=127.5.98.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.98.212:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:32:02.884640 28434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } attempt: 1
I20260924 15:32:02.885111 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.884964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41280)
0924 15:32:02.885043 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.885081 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 15:32:02.885458 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.885359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41286)
0924 15:32:02.885405 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.885437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:32:02.885553 28434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:33371: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
I20260924 15:32:02.889132 28357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } attempt: 3
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:02.896377 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.889472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.5.98.253:34705)
0924 15:32:02.889539 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.889541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.889552 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:32:02.889622 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.889623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.889638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.889706 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.890028 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.890526 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.891247 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.891475 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.891554 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.893707 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.893708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.893708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.893715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.895217 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.895637 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.895638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.895638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.895733 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.895943 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.895943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.895944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.896047 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.896096 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.896257 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.896318 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:32:02.896430 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.889401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43288)
0924 15:32:02.889459 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.889479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.889553 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.889712 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.889713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.889874 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.889941 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.889943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.890609 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.890611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.891159 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.891161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.891552 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.891553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.891634 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.892126 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.892139 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.893599 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.895293 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.895295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.895295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.895549 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.895799 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.895800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.895800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.895876 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.896108 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.896109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.896160 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.896233 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.896332 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:32:02.896867 28357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:02.897202 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.897095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41302)
0924 15:32:02.897142 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.897177 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 15:32:02.897511 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.897419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41314)
0924 15:32:02.897469 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.897494 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:32:02.897604 28357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:33371: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
I20260924 15:32:02.928751 28357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } attempt: 1
I20260924 15:32:02.929263 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.929122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41330)
0924 15:32:02.929185 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.929227 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:32:02.929646 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.929545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41336)
0924 15:32:02.929592 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.929621 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:32:02.929746 28357 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:33371: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
W20260924 15:32:02.941393 28441 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:32:02.941581 28441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:32:02.941598 28441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:32:02.941646 28441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:32:02.942760 28441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:32:02.942798 28441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:32:02.942812 28441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:32:02.942860 28441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:32:02.942884 28441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:32:02.944430 28441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.98.212:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.98.254:44205,127.5.98.253:34705,127.5.98.252:33371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/krb5kdc/kudu_127.5.98.252.keytab
--principal=kudu/127.5.98.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.98.252:33371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.98.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:32:02.944672 28441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:32:02.944895 28441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:32:02.949532 28446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:32:02.949659 28449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:32:02.949751 28441 server_base.cc:1034] running on GCE node
W20260924 15:32:02.949501 28447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:32:02.950790 28441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:32:02.951092 28441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:32:02.952251 28441 hybrid_clock.cc:648] HybridClock initialized: now 1790263922952240 us; error 43 us; skew 500 ppm
I20260924 15:32:02.953685 28434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } attempt: 2
I20260924 15:32:02.954037 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.953920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41342)
0924 15:32:02.953978 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.954009 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:32:02.954358 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.954271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41350)
0924 15:32:02.954313 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.954336 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 15:32:02.954484 28434 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.98.252:33371: Network error: Client connection negotiation failed: client connection to 127.5.98.252:33371: connect: Connection refused (error 111)
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:32:02.954998 28441 init.cc:377] Logged in from keytab as kudu/127.5.98.252@KRBTEST.COM (short username kudu)
I20260924 15:32:02.955421 28441 webserver.cc:460] Webserver started at http://127.5.98.252:42571/ using document root <none> and password file <none>
I20260924 15:32:02.955691 28441 fs_manager.cc:359] Metadata directory not provided
I20260924 15:32:02.955770 28441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:32:02.955917 28441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:32:02.956961 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data/instance:
uuid: "7fb03f43fa5843bd92af615dbd2d15bb"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "24179ac8de3d87c8361a4314229ab1c5"
server_key_iv: "370638414770ba737785efa5bf44dc54"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.957333 28441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal/instance:
uuid: "7fb03f43fa5843bd92af615dbd2d15bb"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "24179ac8de3d87c8361a4314229ab1c5"
server_key_iv: "370638414770ba737785efa5bf44dc54"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.958616 28441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:32:02.959295 28456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:32:02.959465 28441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:32:02.959571 28441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data,/tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal
uuid: "7fb03f43fa5843bd92af615dbd2d15bb"
format_stamp: "Formatted at 2026-09-24 15:32:02 on dist-test-slave-4g6f"
server_key: "24179ac8de3d87c8361a4314229ab1c5"
server_key_iv: "370638414770ba737785efa5bf44dc54"
server_key_version: "encryptionkey@0"
I20260924 15:32:02.959658 28441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:32:02.969441 28441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:32:02.971834 28441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:32:02.972002 28441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:32:02.975955 28441 rpc_server.cc:307] RPC server started. Bound to: 127.5.98.252:33371
I20260924 15:32:02.976004 28510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.98.252:33371 every 8 connection(s)
I20260924 15:32:02.976380 28441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/data/info.pb
I20260924 15:32:02.977340 28511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:32:02.978981 28511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:32:02.983527 5515 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskC7bDtI/build/release/bin/kudu as pid 28441
I20260924 15:32:02.983650 5515 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskC7bDtI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263655576751-5515-0/minicluster-data/master-2/wal/instance
I20260924 15:32:02.983879 5515 external_mini_cluster.cc:1428] Setting key 0e3db0e2f417ade21c30693e08b09bef
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
I20260924 15:32:02.988517 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.979779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.5.98.254:44205)
0924 15:32:02.979862 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.979865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.980052 (+ 187us) server_negotiation.cc:407] Connection header received
0924 15:32:02.980678 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.980679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.980690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.980746 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.981342 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.981815 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.982743 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.982919 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.982970 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.985636 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.985638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.985638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.985646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.987031 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.987456 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.987458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.987458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.987530 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.987847 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.987848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.987849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.987970 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.988043 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.988325 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.988415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:32:02.988659 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.979745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.254:44205 (local address 127.0.0.1:35764)
0924 15:32:02.979924 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.979966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.980571 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.980820 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.980824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.981065 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.981269 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.981274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.981889 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.981889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.982672 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.982674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.982998 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.982998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.983115 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.983626 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.983643 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.985550 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.987135 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.987142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.987144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.987378 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.987610 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.987611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.987611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.987775 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.988051 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.988058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.988190 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.988286 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.988434 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260924 15:32:02.989197 28511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:02.992909 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.984260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.5.98.252:33371)
0924 15:32:02.984432 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.984436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.984491 (+ 55us) server_negotiation.cc:407] Connection header received
0924 15:32:02.984552 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.984554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.984570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.984630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.985372 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.986329 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.987168 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.987463 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.987545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.990091 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.990100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.990100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.990107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.991629 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.992055 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.992057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.992058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.992115 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.992324 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.992325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.992325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.992515 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.992682 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.992729 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.992807 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:02.996367 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.989438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.253:34705 (local address 127.0.0.1:43292)
0924 15:32:02.989484 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.989495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.989529 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.989699 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.989700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.989831 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.989872 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.989873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.990528 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.990529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.991018 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.991020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.991424 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.991425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.991495 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.991869 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.991874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:02.993209 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:02.995345 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.995346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.995347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.995586 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.995825 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:02.995826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:02.995826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:02.995891 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:02.996117 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:02.996117 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:02.996158 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:32:02.996233 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:32:02.996310 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:32:02.996431 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.989528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.5.98.253:34705)
0924 15:32:02.989579 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.989582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.989590 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:02.989615 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.989616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.989626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.989698 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.989949 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.990402 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.991104 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.991348 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.991427 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.993295 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.993296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.993297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.993304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:02.994718 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.995651 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.995652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.995652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.995747 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:02.995953 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:02.995953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:02.995953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:02.996053 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:02.996103 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:02.996269 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:32:02.996342 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:32:02.997076 28511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.254@KRBTEST.COM
Sep 24 15:32:02 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.252@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:03.001524 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.993818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.5.98.254:44205)
0924 15:32:02.993863 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.993865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.994109 (+ 244us) server_negotiation.cc:407] Connection header received
0924 15:32:02.994140 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.994141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.994151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.994225 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.994786 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.995243 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.996244 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.996538 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.996604 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.998740 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:02.998741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:02.998742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:02.998748 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.000380 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.000742 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.000743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.000743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.000802 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.001014 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.001016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.001016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.001132 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.001307 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.001382 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.001460 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:03.003440 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.997304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41370)
0924 15:32:02.997351 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:02.997361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:02.997397 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:02.997508 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:02.997508 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:02.997633 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:02.997656 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.997657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.998257 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.998258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.998821 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:02.998824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.999157 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.999158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:02.999217 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:02.999564 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:02.999569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:03.000910 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:03.002493 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.002494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.002494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.002720 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.002912 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.002913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.002913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.002984 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.003245 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:03.003245 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:03.003277 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:32:03.003323 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:32:03.003386 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:32:03.003549 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:02.997368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.5.98.252:33371)
0924 15:32:02.997403 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:02.997405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:02.997420 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:32:02.997438 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:02.997438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:02.997447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:02.997504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:02.997725 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.998166 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.998906 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:02.999101 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:02.999154 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.000982 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.000983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.000983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.000989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.002432 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.002801 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.002802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.002802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.002849 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.003055 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.003056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.003056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.003182 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.003252 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.003367 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.003440 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:32:03.004879 28357 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } attempt: 2
I20260924 15:32:03.004976 28511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: Bootstrap starting.
I20260924 15:32:03.006064 28511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:03.006384 28511 log.cc:824] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: Log is configured to *not* fsync() on all Append() calls
Sep 24 15:32:03 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.98.253@KRBTEST.COM
I20260924 15:32:03.007285 28511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: No bootstrap required, opened a new log
Sep 24 15:32:03 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.254@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:03.009495 28511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.009656 28511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb03f43fa5843bd92af615dbd2d15bb, State: Initialized, Role: FOLLOWER
I20260924 15:32:03.009752 28511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.010067 28511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:03.010026 28517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.010486 28517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.011189 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.002195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.5.98.253:34705)
0924 15:32:03.002230 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.002232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.002273 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:32:03.002343 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.002344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.002352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.002395 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.002983 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.003698 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.004379 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.004678 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.004755 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.007397 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.007399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.007399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.007405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.009412 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.009831 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.009832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.009832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.009896 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.010292 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.010293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.010293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.010425 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.010760 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.010978 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.011062 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:03.012163 28529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 15:32:03.012593 28360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.005278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41378)
0924 15:32:03.005325 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:03.005335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:03.005373 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:03.005499 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:03.005500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:03.005643 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:03.005674 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:03.005675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.006453 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.006454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:03.007145 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:03.007148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.007636 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.007637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:03.007744 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.008163 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:03.008168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:03.009557 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:03.011445 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.011446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.011447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.011705 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.011920 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.011921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.011922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.011986 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.012268 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:03.012269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:03.012317 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:32:03.012390 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:32:03.012490 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:32:03.012877 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.005328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.5.98.252:33371)
0924 15:32:03.005366 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.005367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.005375 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:03.005432 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.005433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.005441 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.005495 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.005747 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.006362 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.007238 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.007558 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.007626 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.009657 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.009659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.009659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.009666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.011212 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.011789 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.011790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.011790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.011847 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.012055 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.012056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.012056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.012199 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.012274 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.012712 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.012797 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:32:03.012967 5515 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 15:32:03.012216 28529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:32:03.014451 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: Bootstrap starting.
I20260924 15:32:03.014016 28330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "18d3cb819099404f98b78a23dab52182"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35770
I20260924 15:32:03.012215 28441 master_runner.cc:186] Error getting master registration for 127.5.98.254:44205: OK, instance_id { permanent_uuid: "18d3cb819099404f98b78a23dab52182" instance_seqno: 1790263922661615 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:32:03.015074 28441 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:32:03.015208 28357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:03.015439 28357 log.cc:824] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: Log is configured to *not* fsync() on all Append() calls
I20260924 15:32:03.016299 28357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: No bootstrap required, opened a new log
I20260924 15:32:03.017612 28357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.017751 28357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d3cb819099404f98b78a23dab52182, State: Initialized, Role: FOLLOWER
I20260924 15:32:03.017832 28357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.018146 28357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:03.018185 28330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:32:03.018121 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.018247 28330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:03.018324 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.019064 28330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.019311 28330 leader_election.cc:288] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [CANDIDATE]: Term 1 election: Requested vote from peers 830c9dd0aa3644e39fbba5b4318e0676 (127.5.98.253:34705), 7fb03f43fa5843bd92af615dbd2d15bb (127.5.98.252:33371)
I20260924 15:32:03.019708 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18d3cb819099404f98b78a23dab52182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7fb03f43fa5843bd92af615dbd2d15bb"
I20260924 15:32:03.019830 28483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:03.020747 28483 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d3cb819099404f98b78a23dab52182 in term 1.
W20260924 15:32:03.021188 28543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:32:03.021243 28543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:32:03.021255 28307 leader_election.cc:302] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18d3cb819099404f98b78a23dab52182, 7fb03f43fa5843bd92af615dbd2d15bb; no voters:
I20260924 15:32:03.021373 28532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:32:03.021494 28532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [term 1 LEADER]: Becoming Leader. State: Replica: 18d3cb819099404f98b78a23dab52182, State: Running, Role: LEADER
I20260924 15:32:03.021569 28532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.021960 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d3cb819099404f98b78a23dab52182. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.022034 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: This master's current role is: LEADER
I20260924 15:32:03.022759 28288 master_runner.cc:186] Error getting master registration for 127.5.98.253:34705: OK, instance_id { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" instance_seqno: 1790263922808758 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:32:03.025200 28288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:32:03.025826 28553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:32:03.026010 28553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:32:03.028344 28483 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [term 1 FOLLOWER]: Refusing update from remote peer 18d3cb819099404f98b78a23dab52182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:32:03.028792 28532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:32:03.031049 28517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d3cb819099404f98b78a23dab52182. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.031148 28517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.034430 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.020411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.5.98.253:34705)
0924 15:32:03.020479 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.020482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.022712 (+ 2230us) server_negotiation.cc:407] Connection header received
0924 15:32:03.022813 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.022813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.022824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.022856 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.023632 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.024349 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.026732 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.027029 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.027077 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.029636 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.029637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.029637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.029646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.032437 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.032834 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.032835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.032835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.032893 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.033786 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.033787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.033787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.033918 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.034005 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.034237 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.034327 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:32:03.037652 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.037751 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: This master's current role is: LEADER
I20260924 15:32:03.038196 28517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.038278 28517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.038582 28556 mvcc.cc:205] Tried to move back new op lower bound from 7332921028718940160 to 7332921028696850432. Current Snapshot: MvccSnapshot[applied={T|T < 7332921028718940160}]
I20260924 15:32:03.038749 28555 mvcc.cc:205] Tried to move back new op lower bound from 7332921028718940160 to 7332921028696850432. Current Snapshot: MvccSnapshot[applied={T|T < 7332921028718940160}]
I20260924 15:32:03.040297 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.022304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.5.98.254:44205)
0924 15:32:03.022350 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.022353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.027736 (+ 5383us) server_negotiation.cc:407] Connection header received
0924 15:32:03.028845 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.028847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.028856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.028885 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.030055 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.030753 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.033638 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.034222 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.034280 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.036205 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.036207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.036207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.036231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.038922 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.039347 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.039348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.039348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.039408 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.039675 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.039676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.039677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.039804 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.039906 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.040126 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.040203 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:32:03.044341 28553 catalog_manager.cc:1347] Generated new cluster ID: cdf18ea27afc4777a099a37147c8f704
I20260924 15:32:03.044387 28553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:32:03.044903 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.020693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.5.98.252:33371)
0924 15:32:03.020730 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.020732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.027499 (+ 6767us) server_negotiation.cc:407] Connection header received
0924 15:32:03.027582 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.027583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.027595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.027694 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.031565 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.032279 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.033290 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.033587 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.033630 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.037525 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.037526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.037527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.037533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.039058 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.043950 (+ 4892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.043951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.043951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.044015 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.044293 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.044294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.044295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.044427 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.044507 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.044747 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.044821 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:32:03.045616 28434 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } attempt: 3
Sep 24 15:32:03 dist-test-slave-4g6f krb5kdc[28274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263922, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.98.253@KRBTEST.COM for kudu/127.5.98.252@KRBTEST.COM
I20260924 15:32:03.053251 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.045823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.98.252:33371 (local address 127.0.0.1:41396)
0924 15:32:03.045884 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:32:03.045892 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:32:03.045922 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:32:03.046188 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:32:03.046188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:32:03.046325 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:32:03.046361 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:03.046362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.047202 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.047203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:03.047993 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:32:03.047995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.048468 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.048469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:32:03.048520 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.048912 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:32:03.048916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:32:03.050390 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:32:03.052293 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.052294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.052294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.052471 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.052665 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:32:03.052666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:32:03.052667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:32:03.052713 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:32:03.052959 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:32:03.052960 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:32:03.052995 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:32:03.053055 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:32:03.053155 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 15:32:03.053783 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:03.046002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.5.98.252:33371)
0924 15:32:03.046084 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.046085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.046091 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:32:03.046108 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.046109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.046116 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.046154 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.046442 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.047129 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.048089 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:03.048372 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:03.048425 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:03.050487 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:32:03.050488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:32:03.050488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:32:03.050494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:32:03.052209 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.052551 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.052552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.052552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.052611 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:32:03.052782 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:32:03.052783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:32:03.052783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:32:03.052895 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:32:03.052939 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:03.053638 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 15:32:03.053705 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 15:32:03.055373 28434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: Bootstrap starting.
I20260924 15:32:03.056180 28434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: Neither blocks nor log segments found. Creating new log.
I20260924 15:32:03.056501 28434 log.cc:824] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: Log is configured to *not* fsync() on all Append() calls
I20260924 15:32:03.057670 28434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: No bootstrap required, opened a new log
I20260924 15:32:03.059698 28553 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:32:03.059707 28434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.059854 28434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c9dd0aa3644e39fbba5b4318e0676, State: Initialized, Role: FOLLOWER
I20260924 15:32:03.059943 28434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } }
I20260924 15:32:03.060280 28434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:32:03.060351 28406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:32:03.060379 28553 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:32:03.061291 28406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [term 1 FOLLOWER]: Refusing update from remote peer 18d3cb819099404f98b78a23dab52182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:32:03.061503 28407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18d3cb819099404f98b78a23dab52182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "830c9dd0aa3644e39fbba5b4318e0676"
I20260924 15:32:03.062741 28407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18d3cb819099404f98b78a23dab52182 in term 1.
I20260924 15:32:03.063730 28551 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:32:03.063927 28553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182: Generated new TSK 0
I20260924 15:32:03.064092 28553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:32:03.064649 28574 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:32:03.064842 28574 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:32:03.060242 28562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.065032 28562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.065615 28562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d3cb819099404f98b78a23dab52182. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.065701 28562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.065742 28562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.065795 28562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:32:03.079778 28577 mvcc.cc:205] Tried to move back new op lower bound from 7332921028862947328 to 7332921028696850432. Current Snapshot: MvccSnapshot[applied={T|T < 7332921028718940160}]
W20260924 15:32:03.087898 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.086630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.5.98.253:34705)
0924 15:32:03.086667 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.086670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.087239 (+ 569us) server_negotiation.cc:407] Connection header received
0924 15:32:03.087344 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.087346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.087356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.087385 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.087841 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43312: BlockingRecv error: recv got EOF from 127.0.0.1:43312 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:32:03.088856 28515 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.086963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.5.98.252:33371)
0924 15:32:03.087000 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.087002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.088313 (+ 1311us) server_negotiation.cc:407] Connection header received
0924 15:32:03.088447 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.088448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.088458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.088488 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.088799 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41410: BlockingRecv error: recv got EOF from 127.0.0.1:41410 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:32:03.089659 28361 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.088153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.5.98.254:44205)
0924 15:32:03.088188 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.088190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.089054 (+ 864us) server_negotiation.cc:407] Connection header received
0924 15:32:03.089213 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.089214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.089224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.089252 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.089593 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35796: BlockingRecv error: recv got EOF from 127.0.0.1:35796 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:32:03.091753 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.091089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.5.98.253:34705)
0924 15:32:03.091133 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.091135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.091229 (+ 94us) server_negotiation.cc:407] Connection header received
0924 15:32:03.091333 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.091334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.091350 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.091378 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.091709 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43320: BlockingRecv error: recv got EOF from 127.0.0.1:43320 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 15:32:03.094666 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.093915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.5.98.253:34705)
0924 15:32:03.093946 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.093947 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.094201 (+ 254us) server_negotiation.cc:407] Connection header received
0924 15:32:03.094290 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.094291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.094300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.094328 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.094625 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43330: BlockingRecv error: recv got EOF from 127.0.0.1:43330 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:03.098418 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.097676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.5.98.253:34705)
0924 15:32:03.097717 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.097719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.097951 (+ 232us) server_negotiation.cc:407] Connection header received
0924 15:32:03.098044 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.098045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.098055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.098082 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.098363 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43346: BlockingRecv error: recv got EOF from 127.0.0.1:43346 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:03.104125 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.103354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.5.98.253:34705)
0924 15:32:03.103399 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.103400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.103618 (+ 218us) server_negotiation.cc:407] Connection header received
0924 15:32:03.103754 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.103755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.103765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.103793 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.104087 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43356: BlockingRecv error: recv got EOF from 127.0.0.1:43356 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:32:03.114070 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.113163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.5.98.253:34705)
0924 15:32:03.113210 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.113212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.113543 (+ 331us) server_negotiation.cc:407] Connection header received
0924 15:32:03.113643 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.113644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.113654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.113682 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.114015 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43364: BlockingRecv error: recv got EOF from 127.0.0.1:43364 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 15:32:03.132107 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.131215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.5.98.253:34705)
0924 15:32:03.131253 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.131255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.131515 (+ 260us) server_negotiation.cc:407] Connection header received
0924 15:32:03.131618 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.131619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.131630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.131700 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.132058 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43374: BlockingRecv error: recv got EOF from 127.0.0.1:43374 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:32:03.166332 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.165357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.5.98.253:34705)
0924 15:32:03.165406 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.165408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.165705 (+ 297us) server_negotiation.cc:407] Connection header received
0924 15:32:03.165816 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.165817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.165830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.165860 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.166273 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43382: BlockingRecv error: recv got EOF from 127.0.0.1:43382 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:32:03.232476 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.231722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.5.98.253:34705)
0924 15:32:03.231803 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.231804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.231865 (+ 61us) server_negotiation.cc:407] Connection header received
0924 15:32:03.231932 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.231933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.231943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.231991 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.232379 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43390: BlockingRecv error: recv got EOF from 127.0.0.1:43390 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 15:32:03.362630 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.361641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.5.98.253:34705)
0924 15:32:03.361714 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.361716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.361964 (+ 248us) server_negotiation.cc:407] Connection header received
0924 15:32:03.362074 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.362075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.362089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.362174 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.362575 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43404: BlockingRecv error: recv got EOF from 127.0.0.1:43404 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:32:03.544174 28559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18d3cb819099404f98b78a23dab52182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18d3cb819099404f98b78a23dab52182" member_type: VOTER last_known_addr { host: "127.5.98.254" port: 44205 } } peers { permanent_uuid: "830c9dd0aa3644e39fbba5b4318e0676" member_type: VOTER last_known_addr { host: "127.5.98.253" port: 34705 } } peers { permanent_uuid: "7fb03f43fa5843bd92af615dbd2d15bb" member_type: VOTER last_known_addr { host: "127.5.98.252" port: 33371 } } }
I20260924 15:32:03.544431 28559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d3cb819099404f98b78a23dab52182 [sys.catalog]: This master's current role is: LEADER
W20260924 15:32:03.620806 28438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:32:03.620144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.5.98.253:34705)
0924 15:32:03.620216 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:03.620218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:03.620252 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:32:03.620335 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:03.620336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:03.620347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:03.620393 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:32:03.620723 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43412: BlockingRecv error: recv got EOF from 127.0.0.1:43412 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:32:04.013535 28529 catalog_manager.cc:1259] Loaded cluster ID: cdf18ea27afc4777a099a37147c8f704
I20260924 15:32:04.013610 28529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: loading cluster ID for follower catalog manager: success
I20260924 15:32:04.014477 28529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: acquiring CA information for follower catalog manager: success
I20260924 15:32:04.014815 28529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7fb03f43fa5843bd92af615dbd2d15bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:32:04.065304 28574 catalog_manager.cc:1259] Loaded cluster ID: cdf18ea27afc4777a099a37147c8f704
I20260924 15:32:04.065369 28574 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: loading cluster ID for follower catalog manager: success
I20260924 15:32:04.066186 28574 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: acquiring CA information for follower catalog manager: success
I20260924 15:32:04.066536 28574 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 830c9dd0aa3644e39fbba5b4318e0676: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:32:04.136729 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:04.134059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.5.98.253:34705)
0924 15:32:04.134244 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:04.134246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:04.134257 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:32:04.134333 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:04.134334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:04.134386 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:04.134469 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:04.134633 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.135152 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.135969 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.136158 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.136215 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:04.136376 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:04.136489 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:04.136548 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:04.136592 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:32:04.136610 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260924 15:32:04.141048 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:04.138626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.5.98.252:33371)
0924 15:32:04.138781 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:04.138783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:04.138791 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:32:04.138829 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:04.138830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:04.138869 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:04.138944 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:04.139099 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.139596 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.140381 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.140575 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.140633 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:04.140777 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:04.140873 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:04.140928 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:04.140972 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:32:04.140989 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260924 15:32:04.145546 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:04.143068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.5.98.252:33371)
0924 15:32:04.143153 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:04.143155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:04.143162 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:32:04.143183 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:04.143184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:04.143192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:04.143248 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:04.143386 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.143919 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.144777 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.145054 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.145117 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:04.145271 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:04.145302 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:04.145363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:04.145439 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:32:04.145461 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 15:32:04.145737 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:04.142694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.5.98.253:34705)
0924 15:32:04.142738 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:04.142739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:04.142793 (+ 54us) server_negotiation.cc:407] Connection header received
0924 15:32:04.142913 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:04.142915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:04.142924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:04.143019 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:04.143149 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.143911 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.144910 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.145201 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.145277 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:04.145459 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:04.145487 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:04.145529 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:04.145609 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:32:04.145647 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:32:04.146163 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:32:04.142642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.5.98.254:44205)
0924 15:32:04.142868 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 15:32:04.142871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:32:04.143131 (+ 260us) server_negotiation.cc:407] Connection header received
0924 15:32:04.143179 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:32:04.143180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:32:04.143213 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:32:04.143306 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:32:04.143468 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.144222 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.145168 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:32:04.145499 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:32:04.145567 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:32:04.145783 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:32:04.145896 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:32:04.145986 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:32:04.146044 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 15:32:04.146080 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":47,"threads_started":1}
I20260924 15:32:04.147958 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 28288
I20260924 15:32:04.153728 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 28364
I20260924 15:32:04.158872 5515 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskC7bDtI/build/release/bin/kudu with pid 28441
2026-09-24T15:32:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2815 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5600 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268587 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST